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ABSTRACT 

Research has shown that an effective assessment of 
workplace literacy programs requires both formative and summative 
evaluations. Workplace literacy program impact is best measured using 
a mixture of standard assessment tests and custom-designed 
instruments. A broader conception of adult literacy learning can be 
assessed by using interviews and questionnaires to gather information 
about learner literacy beliefs, practices, processes, and plans. In 
assessing the effects of workplace programs on workers’ families, 
five factors have been identified: parents* socioeconomic status; 
educational level; aspirations for their child’s education; ability 
to model literacy practices; and encouragement of literacy practices 
with their children. To assess the impact of a workplace literacy 
program on employer objectives, measures of productivity — company 
records, employee interviews and questionnaires, and supervisor 
ratings — should be taken before and after training. Parallel studies 
of two workplace programs implemented an impact assessment model* 
Findings demonstrated that onsite personnel could perform a 
broad-scale assessment of workplace literacy programs with reasonable 
time frames, using interviews, tests, questionnaires, rating scales, 
and company records. Positive improvements in each area of the 
assessment model — beliefs, practices, processes and abilities, plans, 
productivity, and family literacy — were demonstrated. (Appendices 
include a 79-item bibliography and evaluation instruments.) (YLB) 
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Abstract 

Parallel studies of two workplace literacy programs at 
different sites have been used: (1) to develop an impact 
assessment model for workplace literacy programs, and 
(2) to produce data on the impact of the two quite 
different workplace literacy programs in the areas of 
learner gains, workplace improvements, and literacy- 
related changes in learners’ families. Assessments in these 
two workplaces were used to develop model techniques 
and instruments for impact assessment. The evaluation 
was carried out for each program in conjunction with an 
on-site coordinator who was trained by the principal 
investigator to assist in the collection of data. The 
workplace literacy assessment model focuses on: 

• Learners — changes in beliefs about literacy and 
self, changes in literacy practices, literacy 
improvement with general and workplace 
materials, and changes in goals 

• Employer objectives — improved safety, attendance, 
and productivity, and meeting corporate goals 

• Family literacy — involvement in literacy activities 
with one’s children and changes in home literacy 
practices 

The impact assessment model was used successfully at 
the two sites. It was demonstrated that it is possible for on- 
site personnel to perform a broad-scale assessment of 
workplace literacy programs within reasonable time- 
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frames, using interviews, tests, questionnaires, rating 
scales, and company records. 

Results demonstrated positive improvements in 
each area of the assessment model (i.e., beliefs, 
practices, processes and abilities, plans, productivity, 
and family literacy). However, gains were limited to 
areas directly addressed by instruction, i.e., programs 
and classes accomplished gains only in areas where 
there tvas direct instructional activity. No clear carry- 
over or transfer to other areas was apparent in 
evaluation resulls. There are obvious implications for 
instructional planners — programs need to have 
clearly-stated goals and instruction must address 
those goals if the desired results are to be achieved. 
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Chapter 1 



What We Know About 
Workplace Literacy Programs 



Overview 

Though a growing body of research has identified principles 
and elements associated with effective workplace literacy programs, 
few programs are able to incorporate all elements. Evaluation of 
workplace literacy programs is farther complicated by the fact that 
there appears to be a variety of workplace literacy problems, each 
calling for a different sort of instruction. Still, over the last two 
decades, we have learned a good deal about what to look for in 
effective workplace literacy programs. 

For example we have learned that: 

• There are several different workplace literacy 
problems, calling for a multi-stranded approach to 
instruction. 

• Improvement takes a significant amount of learner 
practice time. 

• Transfer of learning from one application to new 
applications is very limited. 

• Significant learning loss occurs within a few weeks 
if skills are not practiced. 

We have also learned that effective workplace literacy programs are 
characterized by active involvement of project partners (including 
employees) in systematically determining local literacy needs and 
developing programs. 

Multiple Strands for Multiple Problems 

It is important to realize that we face several literacy problems 
in the workplace and not just one. People who can’t read at all 
require different support than do high school graduates who can’t 
meet the new reading demands of their jobs. People educated in a 
foreign language who don't speak much English require another 
sort of support. Providing the same services and programs to such 
different clients makes no sense, and yet it sometimes occurs. 

Increasingly, programs in business and industry are becoming 
multi-stranded. In such programs, one instructional strand might be 
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available to English as Second Language (ESL) learners, while 
other strands are available to learners wishing to pursue GED 
certificates in preparation for further education or high school 
graduates preparing for technical training. Even the format for 
instruction may vary, from structured classes to small group 
instruction to computer guided learning to individual tutoring. 

Bussert (199D sur\'eyed 107 workplace literacy programs 
described in the research literature. Of the descriptions of 
workplace literacy programs providing sufficient information for 
judgments to be made, the vast majority (74%) offered a multiple 
strand curriculum (i.e., two or more of the following: ABE, GED, 
ESL, a selection of basic skills’ technical courses) while 13% 
reported self-pacing of learning (he., home study, PLATO 
computerized learning, learning modules). 

improvement Takes Significant Learner Practice Time 

Training material and technical reading material in the 
workplace tend to range in difficulty, from upper high school to 
beginning college levels (Sticht, 1975; Mikulecky, 1982; Rush, 
Moe & Storlie, 1986). Some learners, such as high school 
graduates who need to brush-up reading skills, can learn to 
comprehend technical materials with a minimum of instruction 
time (about 30-50 hours). Other learners who have extreme 
difficulty with even simple reading, such as signs or simple 
sentences, may require several hundred hours of instruction or, 
indeed, may never be able to comprehend some technical 
material. Gains do not come quickly. Tlie average program takes 
approximately 100—120 hours of practice time for learners to 
make the equivalent of a year gain in reading ability. Auspos, 
Cave, Doolittle, and Hoerz (1989) reported that several hundred 
learners in a pre-work literacy program in 13 diverse sites across 
the country averaged 132 hours of basic education. When the 
participants were tested for reading gains using the Test of Adult 
Basic Education (TABE), an average .7 of a year gain in reading 
ability after approximately 100 hours of instruction was 
demonstrated. 

Targeted programs which use materials learners encounter 
during everyday activities appear to make more rapid 
improvements, but still take from 50-60 hours per grade-level 
gain (Mikulecky, 1989). Sticht (1982) reported that military 
enlisted men receiving 120 hours of general reading instruction 
averaged an improvement of .7 grade levels in reading ability. 
Enlisted men trained with workplace materials in the same 
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amount of time improved 2.1 grade levels when reading work 
related materials . 

Computer learning programs may also cut learning time slightly, 
probably since there is more reading practice and less discussion. 
Haigler (1990) indicated that learners gained an average of 1.26 
years of reading ability in an average of 78 hours of practice using 
computerized lessons in the JSEP job related basic skills program. 
This is equivalent to about 63 hours of practice for a year of gain. 

Yet, linking learning gain to practice time can be somewhat 
deceptive and misleading. A sense of perspective is needed. A gain 
of one year of reading growth in one hundred twenty hours of 
practice is a bargain compared to the experience of the average 
school child, who spends over a thousand hours for a reading gain 
of one year. Furthermore, the more effective workplace literacy 
programs report reducing learning time to 50-70 hours of practice 
for a year of gain. No program, however, has been able to 
consistently improve reading ability from low-level to high school or 
college standards in 20, 30 or even 50 hours. This is important to 
note because in rriany industries the standard training class is less 
than 30 hours. 

The fact that literacy gains usually take more time than is 
typically allocated in workplace training programs presents a 
problem. For gains to occur, more practice time must be found. 
Effective programs demonstrate at least three possibilities for 
increasing practice time. Some programs immerse employees in 
integrated technical/basic skills classes full-time for several weeks 
(see Delco description in Chapter 6). Other programs provide 
sequences of courses allowing learners to move from one course to 
another and eventually to continue learning at technical schools 
and community colleges. A third program type uses workplace 
materials in training classes and thus reaps the bonus of additional 
practice time as learners read these same materials on the job. 

Transfer to New Applications is Severely Limited 

Research indicates that there is a severe limitation on how much 
literacy will transfer from one type of task to other types of tasks if 
the new tasks are not part of the training. Reading the Bible is 
considerably different than reading the newspaper which, in turn, 
differs significantly from the sort of thinking one does while 
reading a manual. ,After reviewing the cognitive research from the 
late 1970s through the 1980s, Perkins and Salomon (1989) 
concluded: 

To the extent that transfer does take place, it is highly 

specific and must be cued, primed, and guided; it 
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seldom occurs spontaneously. The case for 
generalizable, context-independent skills and 
strategies that can be trained in one context and 
transferred to otfier domains has proven to be more 
a matter of wishful thinking than hard empirical 
evidence, (p. 19) 

Consistently during the past decade, literacy researchers have 
reminded us that literacy is not something you either do or do 
not have. It is not even a continuunL What we mean by literacy 
is more accurately described as literacies. There is some degree 
of overlap between being able to read one sort of material and 
being able to read other sorts. The degree of overlap between 
reading a short story, a poem, a lab manual, an equation, a 
computer screen, a census report, and government documents 
may be severely limited, however. We know very little about the 
degree of overlap and the degree of difference among these 
various literacy formats and tasks. 

Evidence suggesting the limited transfer of literacy skills is 
found in the results from the National Assessment of Educational 
Progress, which surveyed the literacy skills of young adults 
(Kirsch & Jungeblutt, 1986). This survey measured the literacy 
abilities c'^ young adults in three different areas: prose, document, 
and quantitative forms of literacy. Correlations among subject 
performances in these three areas revealed limited overlap in 
literacy abilities (i.e., about 25% shared variance). Among those 
surveyed, being able to read a newspaper was only partially 
related to being able to make sense of a document like a chart, 
table, or form. Some literacy ability apparently will transfer. 
Document reading and prose reading, for example, did not seem 
to be totally separated skills. For most learners, however, this 
degree of shared literacy ability appears to be insufficient for 
transfer to occur easily. The hope that teaching someone to read 
a poem will improve that person's ability to read a computer 
screen is probably misplaced. What we want people to be able to 
do, we need to teach them. Some people are able to make great 
transfers from one situation to others. Such people, unfortunately, 
do not appear to be the norm. 

The limitations of literacy transfer have serious implications 
for workplace literacy programs. This is especially true if 
programs attempt to use traditional, school-type materials. Sticht 
(1982) found that general literacy training did not transfer to job 
applications. He now recommends a functional context approach 
which teaches literacy by using the materials with which the 
learner is likely to function on a daily basis. 





TR93-2 



Significant Learning Loss Occurs without Regular Practice 

The problem of lack of transfer is related to the problem of 
learning loss. When people cannot use what they have learned in 
real-world situations, they tend to lose the new skill because they 
lose the chance to practice it. This is important, because new 
knowledge must be used or it is lost. Sticht’s (1982) report of 
military studies indicated that enlisted men improved in literacy 
abilities while they were in general literacy classes but that within 
eight weeks, 80% of those gains were lost. The only exception to 
this finding occurred when job-related materials were used to teach 
literacy abilities. In the latter case, learning gains held up, probably 
because learners continued to practice the abilities they had 
mastered. 

Continuing to practice literacy abilities is very important. It 
means that efforts and resources can be squandered if learners are 
taught with general materials which have no relationship to the 
materials they see daily. It also suggests that the timing of 
workplace literacy training is important. Preparing learners for the 
basic skills demanded by new jobs may be wasted if learners must 
wait several months before they are able to apply and practice their 
new leaiTiing. 

Some programs (Mikulecky & Philippi, 1990) have analyzed 
specific job tasks and developed instructional materials using both 
work and everyday materials. For example, in banking the careful 
reading of withdrawal and deposit slips involves reading, 
computation, and judgment. Similar skills are required at home in 
reading and filling out forms for mail-order catalogs and in paying 
some bills. Instruction that alternates applying the same strategies 
to workplace and home materials offers an increased possibility for 
practice. Data is not yet available on the effectiveness of this 
strategy in stemming learning loss. 

Effective Workplace Literacy Programs 

To be effective, therefore, workplace literacy programs must 
have well designed instruction and they must be flexible enough to 
meet the needs of both differing learners and changing situations 
on the job. The discussion so far has highlighted the importance of 
designing programs which integrate workplace basic skills 
instnjction with several other types of instruction, e.g., technical 
training, ESL training, GED instruction, and low level literacy 
training. It has emphasized the importance of countering lack of 
transfer and learning loss by providing long-term practice with 
materials and activities directly related to the learners’ everyday 
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demands. Additional elements of effective programs are apparent 
as workplace literacy training across the country is examined. 

Salient Features of Effective Program 

A recent study (Kutner, Sherman, Webb, & Fisher, 1991) of 37 
workplace literacy programs funded by the federal government 
identified four key components of effective programs. One of 
these elements related to directly linking instructional materials to 
literacy tasks identified during job analyses. This connection was 
discussed at some length earlier in this chapter. In addition to this 
clear link between instruction and job tasks, effective programs 
were characterized by; 

• Active involvement by project partners 

• Active involvement by employees in determining 
literacy needs 

• Systematic analysis of on-the-job literacy requirements 

Bussell (1991) reported that most workplace literacy programs 
involve partnerships of some sort. Bussert analyzed descriptions 
of 107 U.S. workplace literacy programs and found 92% to 
involve two or more partners. Sometimes the partners were 
multiple unions or multiple businesses; a school and a business; 
or a government agency, a business, and a union. The most 
common types of partnership among the programs she surveyed 
were the following; 

• Employers working with others, 88% 

• Schools (public, community college, and 
university) working in partnership with others, 

51% 

• Unions working with others, 34%. 

Recmitment and retention were reported to be effective when 
each partner played an active role during the early stages of 
program development and a continuing role in supporting 
program goals. Involvement went beyond leaders, however, to 
include learners themselves, who helped gather materials and 
made suggestions for expanding the collection of custom- 
designed materials. It was usually those closest to the job who 
knew what strategies would be most effective in gathering 
information and solving job problems. Active participation of 
partners sometimes meant supervisors and top job performers 
helping to analyze job tasks and suggesting materials and 
approaches whi:h they found effective in preparing new 
workers. 
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A few specific examples of effective workplace programs can 
help illustrate these elements of effective programs. Earlier 
references to the military programs described by Sticht (1982) and 
to the computerized JSEP program described by Haigler (1990) 
touched on these elements. Now we will examine examples 
provided by Hargroves (1989) and Mikulecky and Strange (1986). 

Boston Federal Reserve Bank's Skills Development Center 

Hargroves (1989) described a well-established workplace basic 
skills program in the banking industry. She presented the results of 
a 15 year study which compared Federal Reserve Bank Skills Center 
basic skills trainees to a peer group of entry’' level workers at the 
Bank in terms of: (1) the effectiveness of training in helping under- 
educated youth catch up; (2) retention; (3) job performance; and 
(4) earning power. 

The Bank’s skills development program integrated basic skills 
with clerical training, supervised work experience, and counseling. 
Trainees came into the program because they lacked basic skills 
which were needed in most clerical jobs. Though 50% of the 
trainees had graduated from high school, half read at or below the 
eighth grade level. Two out of three Skills trainees attended long 
enough to complete an extensive class and on-the-job training 
program leading to job placement at the Bank. 

Hargroves gathered information on 207 Skills Center trainees 
from 1973 to 1988 and compared their employment data to that of 
301 Bank employees hired for entry-level positions from 1974 to 
1986. Her results indicated that several months of formal training 
combined with on-the-job experience and counseling enabled 
under-educated youth to catch up to typical entry-level workers. 
Two thirds of the trainees (who would not otherwise have been 
eligible for employment) were placed in jobs. The trainees, on the 
average, stayed longer than their entry-level peers, despite the fact 
that in the late 1980s there was a low unemployment rate and 
ample job opportunities outside the Bank. The majority of Skills 
Center graduates earned as much as tlieir entry-level peers who 
were both more educated and more experienced. 

In summary, the program produced a supply of 
employees who were trained as well or better than 
other new entry-level employees and understood the 
Bank’s employment practices; it also provided 
trainees to departments on short notice for extra 
clerical help. (p. 67) 
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The Hargroves study highlighted several elements key to 
program success: (1) integrating basic skills, clerical skills, work 
experience, and intensive counseling; (2) self-paced and often 
one-on-one instruction focusing on competence; (3) connections 
to community agencies for recruitment; and (4) good 
communications with Bank supervisors in order to develop job 
placements. 

Two Long-Term Integrated Skill Progress 

Mikulecky and Strange (1986) reported on a program to train 
word processor operators and a second program to train 
wastewater treatment workers. Each program involved extensive 
training time. The word processor operators were paid to attend 
between 14 and 20 weeks of training, 40 hours per week. The 
number of weeks was determined by the trainees’ ability to 
function at levels comparable to those of average word processor 
operators who were already employed. The training program for 
the wastewater treatment plant involved 20 full weeks of 
voluntary training which alternated classroom with on-the-job 
training. The word processor training program screened 
applicants and accepted no one who read more than three grade 
levels below the difficulty level of the business materials read, 
typed, and edited by existing word processor operators. These 
materials ranged from high school to college level in difficulty. 
The wastewater treatment training program, on the other hand, 
provided approximately 100 hours of special literacy support for 
the least academically able of its workers. This support focused 
on preparing trainees to use job and training materials which 
averaged from 1 1th grade to college level in difficulty. Employers 
and top-performing workers helped to analyze job tasks and 
provide benchmarks for acceptable performance. 

The average learner in the word processor training program 
reached job-level competence in 20 weeks. Some of the trainees 
were able to find employment in 14 weeks, while a few took 
nearly 28 weeks. The program concluded in the middle of a 
recession, during which one-third of the cooperating companies 
stopped all hiring. In spite of these economic difficulties, 70% of 
the program participants found employment as word processors 
within two months of completing the program. 

The wastewater treatment program focused on the least 
literate 20% of its workers. Nearly one-half passed the technical 
training post tests. The consensus of technical instructors was that 
less than 5% would have passed without the support of the 
literacy program. Of students attending special training sessions. 
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nearly 70% were able to summarize job materials in their own 
words by the end of training. Only about 10% of the learners 
demonstrated gains in general reading abilities, and those were 
students who invested five or more hours weekly outside of class 
on general reading materials. Retention of students receiving special 
basic skills training was higher than that of more able students who 
attended technical training only. 

Conclusion 

No single class or course seems able to meet the demands of 
the diverse populations within a workplace or to provide a 
sufficient amount of instruction to move very low-level , literates to 
the functional literacy levels called for in today’s workplace. 
Multistrand approaches which involve several different types of 
courses and strings of educational experiences leading to long-term 
training goals appear to offer the highest probability of success. 
Such programs need to encourage learners to practice and retain 
new skills by linking training materials to job and home literacy 
demands. The active involvement of workplace partners appears to 
be key to establishing those links and to systematically analyzing 
program effectiveness. Relatively few workplace literacy programs 
meet all these effectiveness criteria, but the degree to which these 
criteria can be accommodated appears directly related to program 
success. 
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Chapter 2 



Methods for Evaluating 
Workplace Literacy Programs 



Overview 

The previous chapter presented examples of effective 
workplace literacy programs and identified key program 
parameters. To evaluate workplace literacy programs effectively, 
two types of evaluation are desirable: formative and summative. 

Formative evaluation of a workplace literacy program takes 
place during begimiing and middle stages of program operation. Its 
purpose is to identify problem areas which can be addressed and 
modified while change is still possible and productive. Formative 
evaluation usually involves the use of interviews, document 
analysis, and observations to determine: 

• The degree to which all involved with the program 
understand and share program goals 

• Whether the resources in terms of personnel, 
materials, learning environment, and learner time 
are sufficient, given current knowledge, to achieve 
the goals 

• Whether the learning processes and methods 
employed appear to be sufficient to accomplish the 
goals 

Summative evaluation usually takes place at the end of program 
operation and is designed to assess how well the workplace literacy 
program has succeeded. Summative evaluation requires gathering 
pre- and post-program data and then analyzing that data. This 
implies using and developing measures directly related to program 
goals. Typical goals for workplace literacy programs include 
improved learner literacy abilities, improved literacy practices at 
work and elsewhere, changed learner beliefs about literacy, self, 
and education, and improved learner productivity on the job. 
Assessment is often accomplished through use of formal 
standardized tests, informally constructed tests related to the 
workplace, questionnaires related to literacy practices, and 
interviews with learners and supervisors. In additiorr, company 
records and ratings on productivity, safety, attendance, and 
enrollment in subsequent classes can expand the evidence available 
for assessing program impact. 
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Current Workplace Literacy Program Evaluations 

Only a few workplace literacy programs described in the 
research literature report rigorous program evaluations, careful 
documentation of learner gains and impacts on productivity, or 
detailed descriptions of effective program practices. Some of 
these programs have been cited in the previous chapter. See, for 
example, Sticht (1982), Mikulecky and Strange (1986), Hargroves 
(1989), Haigler (1990) and Philippi (1988, 1991). These examples, 
however, are atypical. 

Mikulecky and D’Adamo-Weinstein (1991) observed that the 
majority of workplace literacy programs described in the 
available research literature reported no rigorous evaluation data. 
Many programs simply provided superficial information limited to 
surveys of learner satisfaction and anecdotal reports of 
effectiveness. Occasionally a pre- and post-administration of a 
standardized reading test — usually the Test of Adult Basic 
Education (TABE) or the Adult Basic Learning Examination 
(ABLE) — provided an indication of learner gain in general 
reading ability. Only a few evaluations provided follow-up data 
on the impact of programs on learner job performance, retention, 
or earning power. 

Kutner et al. (1991) recently reviewed workplace literacy 
programs funded by the U.S. Department of Education to 
determine effective program elements. The authors examined 29 
of 37 projects funded by the National Workplace Literacy 
Program to determine which programs were effective and 
merited further examination in order to identify components of 
effective programs. The authors reported that: 

Due to the absence of quantitative data necessary 
to identify particularly effective projects (i.e., 
improved productivity, low participant attrition, or 
improved test scores), study sites were 
recommended to OVAE staff. These sites were 
reported by project directors to have a high 
retention rate. (p. 26) 

Even in federally funded workplace literacy programs, in which 
program evaluation was an expectation of hmding, it was not 
possible to find six programs which had been rigorously 
evaluated for effectiveness. Selection of “effective” programs was 
based upon undocumented reports of retention from program 
directors. 
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Formative and Sumtnathre Evaluations 

It is possible to evaluate workplace literacy programs effectively 
using a combination of formative and summative evaluation. 
Formative evaluation of a workplace literacy program takes place at 
the beginning and during program operation. Its purpose is to 
identify problem areas which can be addressed and modified while 
change is still possible and productive. Summative evaluation of 
workplace literacy programs takes place at the end of program 
operation and is designed to assess how well the program has 
succeeded. Assessing summative program impact requires gathering 
pre- and post-program data and then analyzing that data. 

Mikulecky, Philippi, and Kloosterman have performed several 
such formative/summative evaluations using a version of 
Stufflebeam’s (1974) Context, Input, Process, Product evaluation 
model modified for use with workplace literacy programs. In brief, 
the evaluation model employs the use of interviews, document 
analysis, observations, and test data to determine: 

• The degree to which all involved with the program 
understand and share program goals. 

• Whether the resources in terms of personnel, 
materials, learning environment, and learner time 
are sufficient, given current knowledge, to achieve 
the goals. 

• Whether the learning processes and methods 
employed are sufficient to accomplish the goals. 

These three evaluation goals provide information about the 
program in its formative stages. Results can be reported to program 
providers while there is still time to make program adjustments. A 
fourth, the summative evaluation goal of this technique, addresses: 

• What evidence there is that program goals have 
actually been accomplished. 

Formative Evaluation 

A significant portion of the formative evaluation occurs early 
during program planning and operation (i.e., during formative 
stages of program development). Formative analyses usually 
employ interviews, the examination of program documents, and 
on-site observations to focus upon the degree to which program 
goals are shared, the adequacy of resources for achieving those 
goals, and the degree to which program execution appears to 
match stated program goals. 
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Program Goals. Interviev/s, analysis of memos and planning 
documents, and early program observations often reveal that 
significant differences about program goals exist among funders, 
supervisors, instructors, materials designers, and learners. 
Evaluation feedback during early program stages often initiates 
necessary clarification among program planners and participants. 
In some cases, goals are expanded, in some goals are refined, 
and in some new vendors are sought. 

Exanples of interview questions designed to reveal the various 
views of program goals among the participants and leaders are 
provided below. 



Shared Goals 

1 . What do you consider to be the main purposes, goals, 
and objectives of the basic skills training program(s)? 

2. Given the situation you find yourself in, vdiat do you 
think are the most important things for an instructor to 
be doing? 



Additional information can be gathered from published program 
descriptions and from program planning documents. 

Resources. Resources include the expertise of key personnel 
and the availability of instructional space and materials as well as 
the time available for instruction. Early examination of resources 
sometimes reveals that resources are insufficient to accomplish 
goals espoused by program planners. Typical deficiencies are: (1) 
insufficient learner time to accomplish purported goals; (2) lack 
of appropriate learning materials or lack of resources to develop 
custom-designed materials which match workplace literacy 
program goals; and (3) difficulty in finding instructors with 
knowledge or expertise about workplace literacy requirements. 
Information about resources can be gathered by examining 
program facilities and from interviewing key program personnel. 

Examples of interview questions designed to elicit information 
about program resources follow below. 
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Resources 

1 . List training and experience you've had related to this 
job. 

2. What is your assessment of the Allowing: 

Materials 

Facilities 

3. Please describe the following parts of the training 
program; 

Materials for diagnosing and testing learner abilities. 

How a learner's class and out of class learning time 
should be divided. 

How records are Vept and what use is made of records. 



Learning Processes and Methods. As demonstrated in the research 
reported in the previous chapter, literacy improvement takes a 
significant amount of time and general literacy instruction is not 
very effective for workplace applications. Observation of classroom 
instmction, materials, and schedules sometimes reveals potential 
problems with the learning processes and methods offered by the 
program. Examples are: (1) insufficient time for learners to practice 
literacy or too much class time allocated to discussion; (2) teaching 
general reading instruction with school books, off the shelf 
materials, or materials and activities selected because the instructor 
has found them useful in other settings; and (3) little feedback from 
instructors about learner accomplishments (sometimes instructors 
do not or cannot comment upon what individual learners can and 
cannot do). 

Effective programs typically use workplace-related instructional 
activities and real or modified workplace materials. Teachers are 
familiar with job-literacy demands through direct observation or 
documented analyses of the jobs. When instruction using more 
general approaches or materials occurs, the teacher is usually able 
to relate the instmaion to workplace needs. If instruction is not 
related to the workplace, it is because the program has simply 
elected to use a workplace classroom to address general literacy 
goals. In effective programs, no matter what the goal, sufficient 
learner practice time is available to allow reasonable expectation of 
success. Some effective programs even manage to expand practice 
time through homework. 

Examining the processes used in a workplace literacy program 
can be accomplished through classroom observation, examination 
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of learner records and assignments, and interviews with learners 
and instructors. A -great deal can be learned by asking instructors 
and learners to describe how they spent time during the previous 
class period. Information from such interviews can help 
determine if learning activities and time allocation match program 
goals or if learning time is insufficient to meet these goals. 

Classroom observation also provides information on how 
much time both instructor and learners spend in various 
activities. This information can then be analyzed to determine 
whether instructors are allocating time in ways that reinforce 
stated goals and are likely to be productive for learners. A form 
for recording such observational information follows below. 



Classroom Observation 

Time Student Activity Teacher Activity Commenta 

0 
05 




Make note of time spent by students actually reading or doing things. Also note 
time learners spend listening to the Instructor. When learners arc in small 
groups or worlOng Indlvlduafiy should be menUoned. Spe^ rwte should be 
made when the instructor or a student demonstrates how to do something. 



Summafive Evaluation 

While the formative evaluation provides early information 
about the effectiveness of program operation, the summative 
evaluation provides information about whether the program 
achieved its goals. 

Evidence of Goal Attainment. Well-evaluated workplace literacy 
programs gather baseline data before instruction begins. Typically 
data is gathered on the reading abilities, practices, and beliefs of 
learners. In addition, pre-program data is gathered on worker 
productivity or any other goal espoused by the program. Data- 
gathering is accomplished using formal tests, informally 
constructed tests related to workplace expectations, 
questionnaires, and interviews with learners and sometimes 
supervisors. In addition, company records on productivity, safety, 
attendance, and enrollment in subsequent classes can expand the 
evidence available for assessing program impact. 



Such company information establishes a base for later 
comparisons to end-of-program performance. At the end of the 
program, all learners are once again assessed using the same 
instruments. In some cases, it is possible to compare the 
performances of learners in a workplace literacy program to those 
of a control group of comparable employees who haven’t yet been 
able to receive workplace literacy training. To do this, the control 
group takes pre and post assessments which parallel those taken by 
the instructional group. 

Program goals determine the types of information gathered to 
assess program impact. For example, if the program is to improve 
the ability of learners to perform more effectively in quality 
assurance groups, evidence needs to be gathered on such 
performance before and after training. If training is supposed to 
have a positive impact on learner reading habits at home and at 
work, these, too, need to be assessed before and after the program. 

Chapters 3, 4 and 5 will provide samples of methods and 
instruments for assessing the impact of workplace literacy programs 
on learner literacy abilities, practices, plans, and beliefs. In addition, 
methods for assessing the impact of workplace literacy programs 
upon productivity and upon the families of learners will be 
discussed and sample measures will be provided. 

Conclusion 

Only a few workplace literacy programs have been evaluated 
well, even though millions of dollars have been invested in their 
development and operation. To evaluate workplace literacy 
programs effectively it is desirable to perform both formative and 
summative evaluations. Formative evaluation takes place during 
beginmng and middle stages of program operation and is designed 
to identify problem areas which can be addressed and modified 
while change is still possible and productive. The process usually 
involves the use of interviews, document analysis, and 
observations. Summative evaluation of workplace literacy programs 
usually takes place at the end of program operation and is designed 
to assess how well the program has succeeded. It requires 
gathering pre- and post-program data and then analyzing that data. 
This implies using and developing measures directly related to 
program goals. Typical goals for workplace literacy programs 
include improved learner literacy abilities, improved literacy 
practices at work and elsewhere, changed learner beliefs about 
literacy, self, and education, and improved learner productivity on 
the job. 
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Chapter 3 



Assessing Workplace Literacy 
Program Results 



Overview 

The summative evaluation of the impact of workplace literacy 
programs is best performed using a combination of standard 
assessment tools and custom-designed measures. The custom- 
designed measures usually reflect the types of reading done on the 
job and in training courses. In addition, they can focus upon 
special objectives central to the workplace literacy program (e.g., 
increased productivity and comprehending safety information). This 
chapter will discuss several standard and custom-designed 
measures and provide examples. 

Among the topics discussed are: 

• Standardized tests, their advantages and 
disadvantages 

• Custom-designed measures such as Cloze tests and 
job scenarios based on literacy task analysis 

• Assessing a broader conception of adult literacy 
growth which includes learners’ literacy beliefs, 
practices, processes, and plans. 

Model custom-designed measures, including Cloze tests, interviews, 
and questionnaires ane available in Appendices A-C. 

Standardized Tests 

Standardized reading tests are sometimes used in workplace 
literacy programs as a means of identifying the genera/ reading 
abilities of learners. These tests often employ multiple-choice 
questions and short reading passages, from a few sentences to a 
paragraph or two. Some are based on tests developed for use in 
elementary and secondary schools. 

The most commonly used tests are the Test of Adult Basic 
■Education (TABE) and the Adult Basic Learning Examination 
(ABLE). Occasionally a workplace literacy program operating in 
conjunction with a community college may use higher level general 
reading and study skills tests provided by the college. 
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Advantages 



The advantages of standardized tests are two-fold. They can 
provide information on the general reading abilities of potential 
learners. Many community colleges offering technical training 
courses, for example, will not enroll students with general 
reading or computational abilities below the eighth grade level. 
The results of the National Assessment of Educational Progress 
(Kirsch & Jungeblutt, 1986) indicated that approximately 20% of 
American adults read below the eighth grade level — including a 
significant number of adults who graduated from high school. 
In some industries, more than half of the hourly employees 
scored below an eighth grade level. Such individuals are prime 
candidates for basic skills support before and during technical 
training. Sometimes, standardized tests can be used to help 
identify such individuals. 

In addition, standardized tests can be used as program pre- 
and post-assessments to measure gains in general reading 
abilities. Comparison of pre- and post-test scores can indicate 
the degree of effectiveness of a program. Also, post-test scores 
can indicate whether learners are ready to go on to textbooks 
and other general materials in technical training classes. These 
scores are generally indicative of how well someone can 
understand material with which he or she has little familiarity. 
For example, adults scoring at the 10th grade level on a 
standardized test would be very likely to have some difficulty 
with a textbook on an unfamiliar topic which was written above 
the 10th grade level. With some background knowledge on the 
topic, such people might be able to comprehend material a few 
grades above their standardized test scores. It is extremely rare 
that an individual can comprehend material more than a few 
grade levels above his or her standardized test scores (i.e., even 
extensive background knowledge is nearly always insufficient 
to allow a reader at the sixth grade level to comprehend a 
manual written at the llth-12th grade level). 

Disadvantages 

The disadvantages of using only standardized tests in 
workplace literacy programs were introduced in Chapter 1. 
These tests measuie general reading abilities and not the special 
sorts of literacy skills required in the workplace. A learner in a 
general basic skills class may improve in general reading 
abilities. For example, a learner could move from a fifth grade 
level (i.e., understanding the comics and very simple stories) to 
an eighth grade level (i.e., understanding the sports page and 
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USA Today news stories). Though the improved reading ability may 
be of some use, the learner is not likely to be able to transfer those 
skills easily to reading an SPC chart, a technical manual, specialized 
work-orders, and industry-specific textbooks. The most efficient 
way to ensure improvement in these areas is to teach using these 
materials. Unfortunately, gains made in reading job-related materials 
may be only partially reflected in a standardized test which 
evaluates general reading skills. 

More subtle criticisms have been leveled against the use of 
standardized test to evaluate workplace literacy programs. First, the 
information revealed by these tests presents an incomplete picture 
of adult learning (as described above, an adult may read familiar 
materials somewhat better than general test scores indicate). 
Second, the effects of such test scores on instruction are considered 
to be adverse by some (i.e., when teachers teach to the test and 
ignore materials that learners need for the job). Third, the way in 
which standardized test scores are reported can be humiliating to 
adults and counterproductive to learning. Some educators argue 
that when adults are informed that their performance is equivalent 
to a low grade level (i.e., sixth grade or lower), it becomes a 
reminder of their failure rather than an objective description of 
current abilities. 

Recommendation 

Some workplace literacy programs find that the disadvantages 
of standardized tests outweigh the advantages and rely instead on 
interviews, questionnaires, and other indicators to assess program 
effectiveness. Such assessments are custom-designed for the 
program being evaluated. Other programs use standardized tests as 
part of a mix of assessments. If standardized tests are used, they 
should never be the sole measure of learner gain in a workplace 
literacy program. 

Custom-Designed Assessments 

Custom-designed instruments which are based on workplace 
materials and activities can supplement or provide an alternative to 
standardized tests. To design such instruments and, indeed, to 
custom-design training programs, one first needs to determine how 
workers tse literacy in a particular workplace. The first step is to 
perform a literacy task analysis. 
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Literacy Task Analysis 

Literacy task analysis is a way of identifying those aspects of 
job tasks which require reading and problem solving. It is done 
through a combination of observing workers, interviewing top 
performers, and gathering samples of printed materials used in 
the workplace and training classes. The goal is to determine the 
mental processes used by top performers as they solve 
problems and complete tasks which involve literacy. This 
information can be used to constmct both test scenarios and 
instructional materials. It is important that these two be 
developed together so that tests and instructional materials can 
be linked directly to the workplace, and tests can assess what 
learners are really taught. 

Observations of and interviews with supervisors and 
workers are used to identify the areas in which performance 
needs to be improved. Prime targets for analysis are tasks 
where basic skills deficiencies cost money or threaten health 
and safety. Other tasks can be identified by noting changes in 
the workplace (e.g., new technology, changed jobs, or 
promotions) which confront some workers with new and 
sometimes troublesome literacy tasks. 

A good deal has been written about the techniques of 
literacy task analyses (see Mikulecky, 1985; U.S. Departments of 
Education and Labor, 1988; Drew & Mikulecky, 1988; Philippi, 
1988, 1991). Most involve determining the elements of a task 
and the strategies (both visible and mental) employed to 
accomplish the task. For example, filling in forms in some 
quality assurance procedures involves elements such as reading 
two-column charts, computing using decimals, knowing special 
vocabulary and abbreviations, and summarizing sequences of 
events. Within each of these elements, top performers employ a 
variety of strategies (e.g., skimming, estimating, interpolating, 
etc.) 

Philippi (1988) has identified a number of such elements 
and strategies which are listed below. 



Vocabulary 

Recognize common words and 
meanings 

Recognize task related words with 
technical meanings 

Identify word meanings from 
sentence context 



Inferential Comprehension 

Determine figurative, idiomatic, 
and technical meanings of terms, 
using context clues or reference 
sources 

Make an inference from text that 
does not e^mlicitly provide 
required information 
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Vocabulary (continued} 

Recognize meanings of common 
abbreviations and acronyms 

Recognizina cause and 
effect, precucting outcomes 

Use common knowledge to ovoid 
hazard or injury 

Apply preventive measures prior 
to task to minimize security or 
safely problems 

Select appropriate course of 
action in an emergency 

Locating information within 
a text 

Use table of contents, index, 
appendices, glossary, subsystems 
to locate inrermation 

Locate page, title, paragraph, 
figure, or chart nekled to answer 
questions or solve a problem 

Use skimming or scanning to 
determine whether or not text 
contains relevant information 

Cross reference within and across 
source materials to select 
information to perform routine 
activily 

Use a completed form to locate 
information needed to complete a 
task activily 

Comparing and contrasting 

Combine information from 
multiple sources 

Select parts of a text or visual 
materials to complete a task 

Identify similarities and 
differences in objects 



Inferential Comprehension 

(continued) 

Organize information from 
multiple sources into a sequenced 
series of events 

Interpret codes and symbols 

Literal comprehension 

Identify factual details or 
specifications within text 

Follow detailed, sequential 
directions to complete a task 

Determine the essential message 
of a paragraph or selection 

Using charts, diagrams and 
schematics 

Obtain a factor specification from 
a tv'o-column diart to find 
information 

Obtain a factor specification from 
an intersection or row by column 
on a table or chart 

Use a complex table or chart 
requiring cross-referencing within 
text material 

Apply information from tables or 
graphs to locate malfunctions or 
to select a course of action 



Use simple linear path of an 
organizational chart to list events 
in 

a sequential order 

Use the linear path of a flow chart 
to provide visual and textual 
directions for a procedure, to 
arrive at a decision point or to 
provide alternative paths in 
problem solving 

Isolate each major section 
presented in a schematic diagram 



i i 
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Comparing and contrasting 

(continued) 

Determine presence of a defect or 
extent of damage 

Match objects by size, color, or 
significant marking 

Classify objects by size, olor, or 
significant marking 

Distinguish between relevant and 
irrelevant information in texts or 
visuals 



Using charts, diagrams and 
schematics (continued) 

Isolate a problem component in a 
schematic and trace it to the cause 
of the problem 

Interpret symbols to indicate 
direction of flow, text points, 
components, and diagrammatic 
decision points 

Identify details, labels, numbers, 
and parts from an illustration or 
picture 

Identify parts from a key or 
legend 

Interpret drawing of cross-section 
for assembly or disassembly 

Interpret a three-dimensional, or 
exploded view, of an object for 
assembly or disassembly 

Follow sequenced illustrations or 
photographs as a guide 



Materials and information gathered during literacy task 
analysis can be used to develop instructional materials as well 
as to develop custom-designed assessment instruments for 
workplace literacy programs. Examples of such instruments 
(i.e., job-related Cloze tests and literacy scenarios) are discussed 
below. 

Job-Related Cloze Tests 

While standardized tests reveal an individual’s general 
reading ability, a Cloze test is a custom-designed measure to 
assess how well a person can comprehend a particular type of 
reading material, in this instance, job-related information. From 
the workplace materials gathered during the task analysis, 
representative prose passages of about 150 words can be 
selected for the construction of Cloze tests. This is done by 
omitting every fifth word from a passage, usually leaving the 
first and last sentences intact. This results in a passage 
containing about 25 blank spaces which the test-taker is asked 
to fill in, using the surrounding context of sense and grammar. 
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The ability of readers to replace missing words accurately 
correlates very highly with scores on traditional reading 
comprehension tests (Bormuth, 1969; Mikulecky & Diehl, 1980). A 
general rule of thumb is that a score of less than 35% indicates that 
the passage is beyond the comprehension of the test-taker; in other 
words, if the reader can replace less than 9 out of 25 missing 
words, the reading is too difficult. Replacing 50% or more of the 
missing words indicates the ability to read and comprehend the 
material independently. Thus, in a passage with 25 blanks, a score 
of 13 shows that the reading is of a suitable standard for the reader. 
Scores between these values reflect the degree to which the reader 
needs some instructional help to comprehend fully what is being 
read. It is not expected that all the missing words will be replaced 
correctly. A score of 50% is considered quite good, and making 
test-takers aware of this may defuse the frustration they are likely to 
feel when unable to guess satisfactory words for a number of the 
blank spaces. 

A sample Cloze test (with answers) is provided below. The 
instructions include a practice example, since many readers have 
never taken a Cloze test before and sometimes require guidance in 
getting started. Appendix C includes instructions for developing 
Cloze tests am' some additional examples. 



Name pate 

Cloze Exercise 

In a Cloze exercise, you fry to guess which words are missing. For example, in 
the sentence below, a word is missing. 

She looked before she r-^— 

A good guess for the missing word is 'crossed.* 

She looked before she crossed the street. 

In the story below, try to guess and replace the missing words. Don't expect to get 
them all. Some are nearly impossible. 

G.M Designs Safety for All Ages 

We all like to think about the old days. Ufe seemed simpler and, in some ways, 

better then. But when it comes to , the good old days offer the 

satne degree safety as today^s cars trucks. Advancements in 

technology the G.M. vehicle you today among the 

theworld. .Each G.M. and truck is backed 

t housands 

(continues to appmximattly 25 blanks) 



Cloze Exercise key: automobiles, didn't, of, and, make, 
punbase, in 
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Job Problem Solving Simulations and Scenarios 



Simulations and scenarios to assess the job literacy abilities 
of workers can be constructed by using materials from the 
workplace. Information and materials gathered during the task 
analysis form the basis for constructing job-like scenarios in 
which the learner reads and makes decisions based on written 
materials. Scenarios are usually constructed to reflect a range of 
material types (i.e., prose, documents, graphic material), and 
sometimes involve both reading and computation. If the range 
of learner reading abilities is likely to be wide, it is useful to 
construct scenario questions which range from fairly easy to 
fairly complex, so that all test-takers can experience success at 
some level. 

Appendix A contains samples of job scenarios and directions 
for constructing them. For full range testing purposes, it is 
recommended that scenarios include process questions, factual 
questions, inference questions, and application questions. 
Process questions determine how the reader reads a passage, 
i.e., the range and sophistication of reading strategies 
employed. Factual questions are based directly on the reading 
material, inference questions require making deductions from 
several places in the reading, and application questions relate 
the reading to the learner's background knowledge. Examples 
of such questions are provided below. 



Process question 

I am going to show you a newspaper article about your 
industry. 

Explain to me how you would read this story in order to 
find out what the writer thinks. 

Describe what you would look at. What would you be 
thinking about? How would you go about reading this 
story? ’^at would you do first, then next, then next? 

Factual question 

How many employees does ASMO have in Statesville? 
(Answer; 400. Listed in article ) 
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Inference quesHon 

From the information provided about products, what do all 
four companies have in common? 

(Answer: All of them ntKilce some sort of motor. Requires fhe 
interviewee to search for commonalities not readily 
apparent.) 

Applicarion question 

What company makes products closest to your job at this 
facility? 

Why do you say so? 

(Answer: Relate a product on the list to what the employee 
makes. Requires the employee to sort through the information 
and then to apply it to his/her background knowledge. ) 



In addition to their use as a pre-test to establish base-line data 
for assessment, job scenarios can be used to diagnose areas of 
learner difficulty. If the information on the scenarios is also part of 
the training curriculum, the scenarios can provide instructors with 
valuable information. For example, if a learner consistently has 
difficulty with inference questions across ^scenarios, the instructor 
can adjust instruction to provide more guidance and practice in this 
area. The instructor should not, however, provide detailed feedback 
to learners about their performance on the scenarios if the program 
intends to use those scenarios again as a post-test to assess learner 
gain and program effectiveness. 

A test can be used a second time to indicate learner growth if 
the learner has not been taught or given feedback using the actual 
test. It is also important that sufficient time pass between pre- and 
post-tests (six weeks is usually sufficient! If such time is not 
available, it is possible to develop two very similar tests and 
establish the comparability of the two scenarios by noting how a 
pilot group scores on them. This is a fairly lengthy procedure, but 
worthwhile if the tests will be used with many learners over several 
years. Once comparability has been established, the two forms of 
the scenario can be used as pre- and post-measures. However, 
using the same scenarios for both tests provides a more reliable 
means of establishing comparability. 

O 
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Assessing a Broader Conception of Adult Literacy Learning 

Lytle (1990a, 1990b) suggested that performance measures 
(tests and exercises) miss a good deal of important information 
about adult literacy learning. In addition to gains in literacy 
skills, adults may change in what they believe, in how they 
behave, and in their aspirations. Lytle suggests the following 
conceptual framework for a fuller understanding of adult 
literacy and adult literacy growth: learner beliefs about literacy 
and themselves, learner literacy practices, the literacy processes 
employed by a learner while reading, and the plans a learner 
has which may involve literacy use. 

Lytle’s conceptual framework was adapted to the present 
workplace literacy project to test the importance of these 
aspects of adult learning and to seek ways to enhance learning. 
Information about these dimensions of learner literacy was 
gathered through questionnaire items and interview questions. 

Beliefs 

In the interviews, learners were asked to describe 
themselves as readers and writers and to describe someone they 
knew who seemed to be very good at reading and writing. 
They were also asked to provide reasons for their answers. 
Changes in literacy beliefs are likely to precede changes in 
literacy abilities. Sample questions from the interview follow 
below and are also available in Appendix A. 



Beliefs 

1 . Describe someone you know who is good at reading 
and wriring. What makes you choose this person? 

2. How good do you consider yourself to be at reading 
and writing? What makes you think so? 

3. Describe how you would tike to be in terms of 
reading and writing. {Probe : Could you give me 
some examples?) 



Practices 

Learners were asked in the inter\dews and in the 
questionnaire for information about the types of reading and 
writing they do on and off the job. They were asked to rate the 
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difficulty they had in reading each item on a list that included 
books, signs, training manuals, pay stubs, charts, and cartoons. 
They were also questioned about the frequency of their literacy- 
related activities — ^how often, for example, they read a newspaper, 
made a shopping list, or visited a library, as well as how many 
books they owned. Information was also sought about literacy 
practices in workplace situations ranging from departmental 
meetings to handling broken equipment, from reading instruction 
manuals to reading a health insurance policy. 

Sample interview and questionnaire items follow below and are 
available in Appendices A and B. 



Practices 

Interview item 

Tell me the sorts of things you read and write away from 
work during a normal week. (For probe, ask: "Can you 
give me more examples?”) 

Questionnaire items 

1 . First check only the things youVe read in the past 
month.. 

Now go back and rate your ability to read the items 
you've checked. 

poor excellent 



local nev/soaoers 


1 


2 


3 


4 


5 


classified ods 


1 


2 


3 


4 


5 


teleohone bills 


1 


2 


O 

V 


4 


5 


TV auide listings 


1 


2 


3 


4 


5 


rmaazines 


1 


2 


3 


4 


5 


2. In the last 7 days 
newspaper? 


how many 


times have 


you 


read a 


0 1 2 3 4 


5 6 


7 


8 


9 


10+ 



3. You talk a lot in team or department meetings, asking 
questions or sharing ideas. 

very like me 1 2 3 4 5 very unlike me 



3 0 
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Process 

In order to seek information about the processes which 
learners use when reading work materi ils, some questions in 
each job scenario asked students to think aloud about the way 
they were reading the material. The purpose of these questions 
was to determine whether learners were employing 
sophisticated reading strategies (i.e., skimming, focusing, asking 
questions, etc.) and whether the choice and use of reading 
strategies improved as a result of training. Sample questions 
follow below and are available in Appendix A. 



Process 

I am going to show you a newspaper article about 
your industry. 

Explain to me how you would read this story in 
order to find out what the writer thinks. 

Describe what you would look at. What would you 
be thinking about? How would you go about 
reading this story? What would you do first, then 
next, then next? 



Plans 

Interview questions sought information about the learner 
plans, especially in relation to further education and goals 
requiring increased literacy abilities. These questions asked for 
information about learner plans for 1, 5 , and 10 years ahead. 
Sample questions follow below and are available in Appendix 
A. 



Plans 

Now I'd like to ask you about your plans. 

Explain hcrw you see reading and education as part 
of these plans: 

A. *Miat are your plans for the next year? 

B. What are your plans for the next 5 years? 

C. What are your plans for the next 1 0 years? 
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Conclusion 

Workplace literacy program impact is best measured using a 
mixture of standard assessment tests and custom-designed 
instruments. Standardized tests provide useful information about 
general reading ability, but may be misleading with regard to 
workplace literacy skills. 

Custom designing starts with a literacy task analysis to identify 
those aspects of job tasks which require reading and problem- 
solving and where performance needs to improve. Cloze tests 
based on workplace materials can be used to assess workers’ 
abilities at job-related reading. Job scenarios can test their skills in 
using what they read, through process, factual, inference and 
application questions. 

A broader conception of adult literacy learning can be assessed 
by using interviews and questionnaire.^ to gather information about 
learner literacy beliefs, practices, processes, and plans. 



3V 
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Chapter 4 



Assessing Impact on Family 
Literacy 



Overview 

Chapters 2 and 3 have considered the evaluation of 
workplace- literacy programs in relationship to their impact at 
the workplace. Workplace literacy programs also have effects 
on workers’ families and children. 

This chapter considers the factors which can be used to 
measure impact on family literacy. A review of previous 
research on this topic is followed by a discussion of 
questionnaire and interview items used in the current evaluation 
of workplace literacy programs. The complete instruments 
appear in Appendices B and D. 

Topics discussed are: 

• Socio-economic level of parents 

• Education level of parents 

• Aspiration of parents for their child’s education 

• Ability of parents to act as role models 

• Promotion by parents of literacy activities 

Workplace Literacy Programs and Family Literacy 

It is possible for workplace literacy programs to affect not 
only the learners’ literacy levels and productivity on the job but 
also literacy in their families. Home literacy activities can both 
benefit the employees’ children and increase the employees’ 
literacy practice time. Program descriptions provide many 
anecdotal examples of these benefits. A young mother in a 
workplace literacy program at Planters LifeSavers in Virginia 
reported that she enrolled in the company's basic education 
program not only to be able to help her seven children with 
their homework, but also to persuade her oldest son that it was 
important to finish school (Cooper, Van Dexter, & Williams, 
1988). Gross, Lee, and Zuss (1988) reported that one workplace 
literacy student began to help her eight-year-old son with 
homework and was able to leave handwritten messages for her 
children. 
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The effect of literacy programs on the children and families of 
workers is often neglected in evaluating program effectiveness, 
however. At both sites in the current study, assessment of family 
literacy was conducted through pre- and post-questionnaires 
modified from survey questions used by Greer and Mason 0988). 
The questions covered parental guidant:e, literacy artifacts, and 
child-initiated literacy behavior. In addition to individual 
questionnaires for parents, some parents were interviewed in focus 
groups. The Family Literacy Focus Group Interview was 
administered to the participants of the program at only one site and 
was based on the work of Fitzgerald, Spiegel, and Cunningham 
(1991). Sample.? of the questions from each of these instruments 
accompanies the discussion of the impact of parent literacy on 
children which foUov/s. 

Impact Measures 

At least five factors have been ident.ified by research as related 
to the ability of parents to affect a child’s achievement in literacy: 
the socio-economic status of the parents,, their educational level, the 
aspirations they have for their child’s education, the ability of a 
parent to act as a role model, and the parents’ promotion of literacy 
activities. Some of these factors are more easily altered than others 
through a workplace literacy program. The correlation between ihie' 
educational and socioeconomic levels of parents and the child’s 
literacy ability has been identified by researchers as solidly linked 
(Chall, 1984; Laosa, 1984; Sticht, 1983; Sticht & McDonald, 1990). 
However, a brief workplace literacy program is not likely to affect 
income and general education levels directly or very quickly. The 
other three factors are more likely to be affected by a workplace 
program. 

Parents’ aspirations for the best education for their children 
appears to be important in the child's own aspirations, as 
Marjoribanks found (1984a, 1984b). Chall and Snow (1982) showed 
that children whose mothers set high educational goals for them 
achieve higher levels of reading comprehension and word 
recognition. 

Some research indicates that high educational aspirations for 
one’s children may be connected to a parent’s own educational 
level. Laosa (1982) found a significant relationship betwee.n a 
mother’s educational aspirations for her child and the level of 
schooling of both parents. However, Lujan and Stolworthy (1986) 
found that the educational aspirations of lower socio-economic 
status families were just as sincere and ambitious as those of 
parents from middle to higher levels. Unfortunately, as important as 
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aspirations may be, parents who are unable to help their 
children reach such goals are at a disadvantage. 

The ability to model reading and to engage in interactions 
with a child which encourage and teach literacy is important. 
However high the aspirations of a parent might be, illiterate 
adults cannot model what they do not know (Nickse, Speicher, 
& Buchek, 1988). In interviews with parents, Fitzgerald et al. 
(1991) found that low-literacy parents did not even mention 
adult role modeling as important in helping their children, 
whereas high-literacy parents talked about the need to have 
their children see them reading. Work with middle school 
students by Fielding, Wilson, and Anderson (1986) showed that 
student readers tended to have parents and siblings who read. 
A parent’s ability to model oral language skills also seems to 
affect a child’s ability to read in school (Sticht, 1983; Loban, 
1964; Chall & Snow, 1982). 

In the current study, questionnaire and interview items were 
developed to measure effects in these areas. Examples follow 
below. 



Quesrionnaire item 

In the last 7 days how many times has your child seen 
you reading or writing? 

01 23456789 10+ 

Interview item 

At home, do your children see you doing ony reading or 
writing? (i.e., books, magazines, papers, recipes, 
directions, letters, lists, notes, etc.) 



Closely related to the parent as a role model is the activity of 
a parent to encourage a child in literacy activities. Included in 
such activities are the creation of a literacy environment in the 
home and the use of a community library. According to Fielding 
et al. (1986) readers in middle schools come from homes in 
which there are many books and many opportunities to go to a 
library. Similarly, Greer and Mason (1988) found that the 
children who score higher on tests of reading recall are those 
v/ho frequent a library, have someone at home who reads to 
them often and helps them read, and have books and 
magazines purchased for them. 
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Parents who directly promote their childien’s reading have 
children who seem to do better in school. McCormick and Mason 
(1986) sent home easy-to-read Little books for parents to read to 
their preschool children. The parents were given instructions in 
helping their child learn to recite. That activity had a significant 
effect on the children’s later reading in kindergarten and first grade. 
Furthermore, Chall and Snow (1982) discovered that reading 
comprehension was higher for the second, fourth, and sixth graders 
they studied whose homes provided more literacy experiences and 
reading materials which were both interesting and appropriate for 
the child. Stewart (1986) administered a reading test to 56 children 
and compared their scores to the answers their parents had given 
to a questionnaire that assessed home support for early reading. He 
found a significant relationship between borrowing books from a 
public library and the children’s performance on the test. 

Literacy environment in terms of reading materials available in 
the home or trips to the library was assessed in the following 
questionnaire items. 



1 . In the last month how many times hove you bought or 
borrowed books for your child? 

01 23456789 10 + 

2. In the lost month how many times has your child gone to a 
public library? 

01 23456789 10 + 



Serving as a role model and providing materials are not the only 
ways parents improve children's literacy. Activities in which parents 
and children interact together are also important. Such activities 
include reading aloud to a child, encouraging the child to ask 
questions and make predictions about the text, allowing the child 
to initiate a literacy event, and parental involvement with the 
school. Both questionnaire items and questions from the Family 
Literacy Focus Group Interview address such activities. 

Time spent reading with a child, particularly prior to the school- 
age years, can affect the child's later success or failure in reading. 
Stewart (1986) visited the homes of four children several times over 
a two-month period and learned that stimulation from parents 
made more of an impact on children's reading abilities than merely 
having books around the house. In fact, the effect of reading aloud 
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to children has been widely studied. Chomsky (1972) revealed 
that the most important activity for building the knowledge 
required for literacy success is reading aloud to children. Laosa 
(1982) found significant correlations between mothers who read 
to their children and the child's literacy skills in preschool. 
Studies by Buchanan-Berrigan (1989), Anderson (1985), Teale 
(1984), Teale and Sulzby (1986), and Fitzgerald et al. (1991) also 
indicated that reading aloud to children, especially when they 
are active participants, helps in the development of preschool 
literacy, which, in turn, enhances school learning. 

Below are questionnaire items which assess such activity: 



1 . In the last 7 days how many times hove you read/looked at 
books with your child or listened to him/her read? 



01234567 89 10+ 

2. In the last 7 days how many times have you helped your child 
with homework and/ or with school projects? 



01234567 89 10+ 



Several studies have revealed that parents who read to their 
young children also encourage them to label pictures, ask 
questions, and relate text information to their own experiences 
(DeLoache & DeMendoza, 1985; Harkness & Miller, 1982; Snow 
& Ninio, 1986; Pellegrini, Brody, & Seigel, 1985; Yaden, 1982). 
As Mason and Stewart (1988) suggested, these parents are 
leading their children towards the use of inference and 
comprehension monitoring strategies. The benefits of reading 
aloud to children, therefore, seem to be greatest when the child 
is an active participant who engages in discussions about 
stories, learns to identify letters and words, and talks about the 
meaning of words (Anderson, 1985). 



An interview question assessing such interactions follows below: 



Do you do any reading or writing activities with your children? (i.e., 
visit library, hear stories, read to them, watch educational television, 
look at magazines or books with children, point out words to them, play 
school, show them how to read or write, etc.) 



From reading aloud and encouraging the child’s interaction with 
the text, a next step is to attend to whether the child ever initiates 
the reading activity. McCormick and Mason (1986) found that 
parents who were provided with inexpensive books for their 
children reported sigriificantly more child-initiated use of books and 
child-initiated attempts to print than did a control group who were 
not given books. More importantly, the children had invited their 
parents into literacy activities, such as asking to read stories to their 
parents and asking for help with new stories, to a greater extent 
than the children of the second group. Teale (1983) discovered that 
as children become more adept, they take over more and more of 
the interaction until they can read the book alone or write on their 
own without help. 

Child-initiated behavior was more thoroughly examined by 
Lujan and Stolworthy (1986), who found that the most significant 
result from parent training was a positive change in most children's 
literacy behavior. For example, the children began to attend more 
closely to story time and parent instruction. They showed increased 
self-direction in organizing personal time so that there would be 
time at night for story reading. 

Questionnaire items addressing these issues follow below. 



1 . In the last 7 days how many times has your child looked at or 
read books or magazines? 



01 23456789 10+ 

2. In the last 7 days how many times has your child asked to be 
read to? 



01 23456789 10+ 

3. In the lost 7 days how many times has your diiid printed, made 
letters, or written? 



01 23456789 10+ 
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Conclusion 

Workplace literacy providers want to get the most for their 
investment. Effective programs may be able to improve the 
abilities of workers on the job as well as to benefit children in 
the home. Longer term effects of increasing workers’ literacy 
abilities can include examining the effects on a worker’s family 
and children, yet workplace literacy program evaluations often 
neglect such impacts. We know, too, that as workers are 
encouraged to carry newly-won literacy abilities home, they 
benefit from the opportunity to increase their own practice of 
these skills. 

In assessing the effects of workplace programs on workers’ 
families, five factors have been identified. These are: 

• Socio-economic status of the parents 

• Parental educational level 

• Parents’ aspirations for their child’s education 

• The ability of parents to model literacy practices 

• Parental encouragement of literacy practices with their 
children 

The first two are not as readily affected by short-term workplace 
programs and, therefore, are less desirable assessment targets. 

Measurement of parental aspirations, modeling, and 
encouragement were conducted during the current study 
through questionnaires given before and after the program and 
through Family Literacy Focus Group Interviews. 





40 



TR93 



Chapter 5 



Assessing Impaq on Productivity 



Overview 

A review of the literature on productivity assessment shows that 
little is known about the effect of workplace literacy programs on 
job performance, but there is some evidence of the value of such 
programs and of the costs associated with lack of training. 

There are methods to assess the impact on productivity of 
workplace literacy programs. A program can be assessed using 
employee output and such indicators as safety, absenteeism, and 
retention, with these measures taken both before and after training. 
Also, employees can be rated by their supervisors on various 
aspects of job competence and attitude, and changes in these 
ratings can be used in the calculation of the dollar value of the 
program to the company. 

Such methods and others directly related to literacy were 
incorporated into this study. To assess changes produced by a 
program, the following measures were used both before and after 
training: 



• Records of absenteeism, safety, discipline, 
grievances, and suggestions were used to assess 
employee performance. 

• Interviews and questionnaires were used to assess 
job-related literacy practices and processes of 
employees. 

• Supervisor ratings on various aspects of employee 
job competence and attitude were obtained. 

Literature on Productivity Assessment 

Workplace literacy programs have been offered by many 
organizations, both government and private, but not much is 
known about the effect of such programs on the job performance 
of the employees involved. For the most part, the organizations 
have regarded literacy programs more as philanthropic than as 
business enterprises and so have not considered it appropriate to 
subject them to their usual cost-benefit analyses. 

There are, however, a number of indications that such programs 
can have a positive influence on the effectiveness of the workers 
involved. Collino, Aderman, and Askov (1988, p. 19, note 17) 
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mention a Blue Cross/Blue Shield program that decreased 
turnover and improved performance and promotion prospects 
as well as increased motivation and self-confidence among 
employees. Also, the Federal Reserve Bank’s Skills Development 
Center has had considerable success in training under-educated 
school dropouts up to a standard of job performance 
comparable to qualified entry-level workers (Hargroves, 1989). 

Collino et al. (1988) cited a number of cases of the costs 
associated with employees’ lack of basic skills: 

• The inability to read a ruler wasted $700 worth 
of steel in one morning. 

• The inaccurate use of new scheduling 
equipment cost $1 million to correct the 
resulting errors. 

• Employees at a lumber camp imitated the 
illustrations on safety posters because they 
could not read the text deccribing these as 
dangerous practices to be avoided, (pp. 11-12) 

However, the fact remains that there has been very little 
systematic evaluation of workplace literacy, even of its effect on 
employees’ more general ability to cope with everyday literacy 
demands. So it is perhaps hardly surprising that a recent report 
by the U.S. Departments of Education and Labor (1988) should 
conclude: 

Very little research exists about the 
relationship of literacy to job performance. 

Much of what exists is sketchy and based 
on information obtained from studies 
conducted in the military, (p. 37) 

Collino et al. (1988) found that, even when companies do 
conduct assessments of their literacy programs, the results are 
not made public. Furthermore, such assessments rarely involve 
a study of how productivity might be affected. They reported, 
“At best management relied on informal feedback of supervisors 
regarding employee performance.” (p. 9) 

Methods of Use for Workplace Literacy Programs 

A workplace literacy program should have a positive and 
measurable impact on productivity. However, most companies 
do not have an evaluation methodology and therefore can not 
easily recognize the impact on productivity of training workers. 
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Impact on ProdueHvity 



Though little research exists on methods to assess the impact on 
productivity of workplace literacy programs, more research and 
discussion are available on the general topic of the impact of 
training upon productivity (National Research Council, 1979). When 
workers are producing an actual physical output, the quantity or 
quality of that output can be measured before and after training, or 
a comparison can be made between the output of trained and 
untrained workers. Programs that make such assessments are 
usually broad range training programs which can compare the 
output of a trained plant, division, or work team to a comparable 
control group. Assessing productivity impact at levels below the 
work-team is often precluded because many industries do not 
collect productivity information (i.e., production and defect rates) at 
the individual level. 

A broader definition of productivity allows for some information 
to be collected at the individual level. For example, other factors 
that may be affected by a training program are 

• Retention and promotion 

• Absenteeism and punctuality 

• Dishonesty 

• Accident rates 

• Use of suggestion boxes 

In addition, if productivity is broadly defined as supporting 
corporate goals, increased participation in voluntary activities (e.g., 
additional training or employee quality participation groups) can 
also be included among productivity indices (see, for example, 
Collino et al., 1988; U.S. Departments of Education and Labor, 
1988). All of these factors can be used to compare employees 
before and after a program and those employees with others not 
attending the program. 

Supervisor Ratings 

Another way of obtaining information about the effect of 
training on individual workers is to use supervisoi ratings. These 
can be a single score for each employee or, preferably, a set of 
scores covering a variety of specific skills and attitudes associated 
with job performance. Depending on the nature of the work 
concerned, these aspects are likely to include: 

• Setting up and operating machines 

• Keeping up-to-date with paperwork 
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• Taking responsibility for one’s own work 

• Having the initiative to solve problems as they occur 

• Communicating with other workers 

• Being committed to company goals 

For each aspect, a rating scale can be set up with 
descriptions of worker performance at low, average, and high 
levels. For example, in order for supervisors to rate workers’ 
initiative in dealing with machine errors on a scale from 1 to 10, 
the descriptions might be: 



roHng of 2 • ignores machine errors and lets them build up 
rating of 5 • realizes machine errors and attempts solution 
rating of 8 - monitors machine errors and deals with them 



These descriptions anchor the rating scale to specific worker 
behaviors in order to produce consistent ratings both between 
supervisors and from the same supervisor in pre- and post- 
training assessments. Developing the descriptions with the help 
of workers and supervisors enables them to be a realistic 
reflection of job practice. For examples of supervisor ratings, 
see Appendix G. 

Such job performance scales anchored to validated 
behaviors have proven to be useful in lowering error, increasing 
reliability, and being efficient in terms of job performance 
ratings (Borman, 1977; Latham, Wexley, & Pursell, 1975). Job 
performance scales anchored to behaviors have proven to be 
most effective when special care is taken in describing the job 
dimensions to be evaluated (Dickinson, 1977) and when 
unambiguous anchor descriptions are developed with 
involvement from job incumbents and the supervisors who are 
to participate in rating job performance (Norton, Balloun, & 
Konstantinovich, I960). Mikulecky and Winchester (1983) and 
Mikulecky and Ehlinger (1986) have successfully used such 
anchored supervisor ratings to assess job performance in the 
nursing profession and the electronics industry. 

An alternative approach is to use an overall assessment of 
the performance of each employee, as rated by their 
supervisors, to calculate the utility of the training or literacy 
program in terms of its benefits minus its costs (see Sheppeck & 
Cohen, 1985; Schmidt, Hunter, & Pearlman, 1982; Cascio, 1982.) 
For this calculation, the factors required are an estimate of the 
difference in dollar value to the company between an 
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outstanding and an average employee, the likely duration of the 
training's effect, and the cost of the program. (See the Endnote to 
this chapter for an example of a utility calculation and further 
details concerning the use of this method.) 

Methods Incorporated Into This Study 

For this study, indicators relating to productivity were gathered 
on each employee both before and after the training program or, in 
the case of on-going programs (such as GED classes), at suitably 
spaced intervals. These included statistics on attendance, safety, 
and suggestions. Interviews and questionnaires assessed employee 
attitudes to the workplace and various job-related skills. Also, 
supervisors assisted in the development of anchored rating scales 
which they then used to assess each employee before and after 
training. 

PmducHviiy Data 

It was not possible, for different reasons at the two companies 
participating in this project, to obtain data on the actual output of 
the individual employees involved in training. Since companies do 
not gather output data for units below that of the work-team, the 
attempt was made to have a whole team at each site take part in 
training at the same time. However, at one site the class could not 
in the end be arranged in that way, as teams were reorganized and 
some individuals could not be released for training. At the other 
site, although all members of a team did go through training 
together, output data for that tea,m could not be separated out from 
plant-wide figures for individual analysis. Thus, in order for the 
gathering of output data to be successful, it must be possible for a 
company to arrange training for a whole work team and for 
mechanisms to be put in place, perhaps especially for this purpose, 
to obtain the output data for that team. 

The following measures were, in fact, used to evaluate changes 
in employee performance, each measure being taken both early 
and late in each program so as to assess the impact of that program 
on the employees involved. In addition, in one case comparisons 
were naade with a control group of employees who had not yet 
participated in the programu Data relating to employee attitudes 
were collected on: 

• Absenteeism 

• Grievances submitted 

• Discipline records 

• Workplace safety records 

NATIONAL CENTER ON ADULT LITERACY 

4 ^ 



45 



• Suggestions made 

• Suggestions accepted 

Interviews and Questionnaires 

To supplement the company records, employees were 
interviewed and also filled out questionnaires (see Appendices 
A and B). Their purpose was, in part, to assess attitudes toward 
the workplace and competencies associated with the workers’ 
jobs. In the interviews, the employees were asked about the 
types and amounts of reading and writing they do on the job to 
assess the quantity and quality of their workplace’ literacy 
activity. They were also asked to demonstrate specific skills in 
using items related to their work, such as job aids or written 
information sheets and graphs or tables. Questions here were of 
two types: process and content. Process related to how the 
workers use the item. For example: 

• Do they use job aids regularly? 

• What parts of them do they look zt? 

• How long does it take them to read one? 



Content questions were more specific to the particular item, 
asking for information that the workers should be able to obtain 
from the sheet in front of them, such as: 

• What components do you need to make this parti* 

• How do you carry out this procedure? 

• What does this graph show as the inventory value on 
a certain date? 

Some content questions called for interpretation by the 
interviewee, drawing on the given information to make 
inferences about the situation. For example: 

• Why do you think the value fell during this particular 
month? 

• What might have caused this type of wastage to occur? 

The questionnaire dealt, in part, with reading and talking in 
relation to the workers' jobs, particularly their abilities and 
confidence in reading instructions and talking in meetings. 
Among items of a more general nature, they were asked to rate 
as easy- or hard-to-read work-related: 
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• Job aids 

• Part specifications 

• Safety rules 

• Benefit information 

• The plant newspaper 

In addition, they were asked to use a scale from “very like me” to 
“very unlike me” in rating such statements as; 

• Your ideas are often discussed in meetings. 

• When written information is handed out, you read 
it to see what it is about. 

• When paperwork comes to you about your job, 
you often have trouble reading it. 

Supervisor Ratings 

To obtain another perspective on the information gathered 
directly from the employees, supervisors were asked to assess each 
worker on aspects of job performance that contributed to 
productivity and that were related to task competence, 
communication, teamwork, and paperwork skills. Assessment 
instruments were developed with the assistance of those who 
would be using them to determine what aspects should be covered 
and how to describe behaviors typical of top, average, and bottom 
performers. Specific aspects included were the ability to; 

• Set up and calibrate a machine 

• Use recording forms 

• Trouble-shoot machine errors 

Also assessed were attitude indices such as; 



• How much they took responsibility for their own work 

• How well they worked as a member of a team 

• How committed they were to conipany goals 

For each of these indices, anchoring descriptors for bottom, 
average, and top performance were related to a scale of from 1 to 
10 to guide the supervisors in making their assessments. 

Thus, the final supervisor rating form could contain instructions 
such as; 
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• An average employee would be rafed 5. 

• A top employee would be rated 8 or higher. 

• A bottom employee would be rated 2 or bwer. 



One item on a form could appear as follows (see Appendix G 
for further examples). 



Paperwork 






Bottom 


Average 


Top 


intimidated 

nil 


does job-related 


completes 


Uil 

by job-related 


paperwork. 


job-related 


paperwork and 


simply 


paperwork 


does it poorly 
to 


keeping pace 


and tries 


procedures. 




improve 


0 12 3 


4 5 6 7 


8 9 10+ 



Conclusion 

In order to assess the impact of a workplace literacy 
program on employer objectives, measures of productivity 
should be taken before and after training. Such measures 
include company records, employee interviews and 
questionnaires, and supervisor ratings. 

Company records can supply information on output, safety, 
dishonesty, discipline and grievances, absenteeism and 
punctuality, retention and promotion, and productivity 
suggestions. 

Employee interviews and questionnaires can supply 
information on attitudes arid job practices and skills. These 
include how much reading and writing employees do in the 
workplace, how competent they are at various types of reading, 
and tfieir confidence with reading and in meetings. 

Supervisor ratings can also supply information on employee 
job-related skills and attitudes. Using anchoring descriptors for 
top, average and bottom performers, rating scales can be 
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developed to cover such aspects as task competence, 
communication, team work and paperwork skills. 



Endnote - Calculation of L/t/7/iy of Training. 

ColculoHon of the utility or cost effectiveness of a training program requires: 

1 . An overall measure of the job performance of each employee trained and of a 
comparable group of untrained workers. (This could be either a supervisor rating 
or be based on production outcomes.) 

2. A measure of the dollar value to the company of the difference between 
outstanding and overage employees. (This estimate of the standard deviation of 
performance is known os the value.) 

3. The expected duration of the training's effect. 

4. The cost of tlie training. 

^ an example of a utility calculaHon, let us suppose that the 20 employees 
who have completed a training program are rated by their supervisors, on average, 
at 65 out of 1(W. The untrained employees received an overage of 50. with a 
standard deviation of 10. The trained workers are at a level of 1 .5 stanoardized 
units above the untrained -- this is the performance difference. If it is estimated that 
*he werage employee is worth $1 8,0C)0 to the company and an outstanding one is 
wom $26/000/ then an estimate of the vo/ue or standard deviation of employee 
performance is $8,000 (the difference between these two amounts). Suppose also 
that training costs $2,000 per employee and the effect of training is likely to last 3 
years. Then we hove: 



Yeofs duration x Number x Performance x Value - Number x Cost per 
or effect trained difference trained trainee 

“3 X 20 X 1 .5 X $8,000 - 20 x $2,000 

- $720,000 - $40,000 

“ $680,000 rtet utility to the company 



This formula was originally developed by Brogden (1 949) and revised into its 
present form by Schmidt, Hunter, McKenzie and Muldrow (1979). For examples of 
its use. see Sheppeck and Cohen (1985), Schmidt, Mack and Hunter 1984), 
Schmidt, Hunter and Pearlmon (1 982) and ^scio (1 982). 

Criticisnris of the method are to be found in Boudreau (1 983) and Cronshow 
and ^exander (1985); these are refuted by Hunter, Schmidt, and Coggin (1988). 
Modrhcations to the procedure are suggested by Bobko, Karren, and Parkington 
(1983) and Cascio and Ramos (1986); see also Cascio (1982). Comparative 
studies of such modifications are contain^ in Greer and Cascio (1 987), Burke and 
Frederick (1 986), and Weekley, Blake, O'Connor, and Peters (1985). 
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Chapter 6 



Structure of Current Study 



Overview 

The purposes of this study were to develop an impact 
assessment model for workplace literacy programs and to produce 
data on the impact of programs at two sites. A secondary goal was 
to refine the model for use at other sites. 

The two sites chosen are very different, but both operate 
established, effective, and diverse programs involving technical and 
communications training and GED and ESL classes. 

Pre- and post-program data were gathered on learners’ job 
productivity^, literacy attributes, and literacy practices in their 
families. Instruments and methods used to gather data included: 

• Interviews, tests and questionnaires based on the 
adult literacy model — beliefs, practices, process, 
and plans — developed by Lytle (1990a, 1990b). 

• Questionnaire items based on key practices for 
developing home literacy. 

• Productivity^ indicators such as attendance, safety, 
and supervisor ratings of on-the-job use of literacy 
and communication skills. 

The data were analyzed using statistical comparisons of 
quantitative information, as well as qualitative and quantitative 
analyses of categories emerging from open-ended responses to 
interview questions. 

Purpose 

Although federal and private support funds thousands of 
workplace literacy programs, very few programs have been 
evaluated beyond a superficial level (Mikulecky & D’Adamo- 
Weinstein, 1991). Typical workplace literacy program evaluations 
involve anecdotal reports, learner satisfaction questionnaires, or 
pre- and post-results from a standardized basic skills test such as 
the TABE or the ABLE. 

In late 1990, the National Center on Adult Literacy funded a 
project to develop and pilot a model for evaluating the impact of 
workplace literacy programs. During Year 1, parallel pilot studies of 
two workplace literacy programs were used to: 
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• Develop an impact assessment model for 
workplace literacy programs 

• Produce data on the impact of two quite 
different workplace literacy programs 

The goals of the first year's efforts were to refine the impact 
evaluation model so that it could be transferred to additional 
sites during subsequent years and to establish base-line data for 
the level of impact one could expect from established 
workplace literacy programs. 

Populations 

The evaluation model to as.sess the impact of workplace 
literacy programs was piloted at two sites: Delco Chassis of 
Rochester, New York, and Cumberland Hardwoods of Sparta, 
Tennessee. Though the sites were chosen for their differences — 
in size, demographics, location, and industry — each site had a 
well-established workplace literacy program which addressed 
several different populations (e.g., technical communication and 
basic skills training, GED preparation, and ESL preparation at 
Delco). Leaders at both companies saw it as necessary for 
survival to increase employee involvement in the decision- 
making processes of day-to-day business. For example, each 
firm intended that those actually producing the goods be able 
to decide whether machines required adjustment or whether 
their production line had stockpiled a sufficient quantity of 
product X and should switch to product Y. 

Both companies had education programs judged by state 
and federal acknowledgment to be effective models of 
workplace literacy education. Since new instruments to assess 
the achievements of such programs were to be piloted in tfie 
study, benchmarks could be established with programs which 
had been independently judged to be good. 

Classes and individuals at each site provided information 
through interviews, tests, checklists, and questionnaires to 
assess the impact of programs upon learners, their productivity, 
and family literacy in their homes. In addition, curriculum 
materials were examined and classroom instruction was 
observed. 
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Subiects and Locations: Delco 



Site #1, Delco Chassis, is a large, unionized (International 
Union of Electrical Workers, Local 509), electrical motor 
manufacturing plant with over 3,600 employees located in 
Rochester, New York. Employees are enrolled in an education 
program jointly operated by union and management in conjunction 
with state and regional agencies that provide some funding and 
help in providing instmctors from the local school system. In this 
study, all learners were from the production teams who participated 
in one of three types of classes: 

• A Technical Preparatiorh—z 6 weel. 7 hours per 
day course, designed to prepare employees for 
subsequent training, which met seven hours per 
day for six weeks 

• A GED preparation course which generally met 
four hours per week in slightly varying time frames 

• An ESL course which met eight hours per week 

In addition, there was a control group for the Technical Preparation 
course composed of workers who had not yet begun classes. Each 
of these four groups consisted of 12-15 employees. There was an 
additional small control group (of five) available for the ESL group. 

The Technical Preparation course was designed to prepare 
learners for the mathematics, reading, oral communication, and 
blueprint reading skills judged to be prerequisite for further 
technical training. Readings and activities were a mixture of some 
plant-specific materials and carefully selected off-the-shelf materials 
related to course objectives. Activities in the reading component of 
the course included study skills exercises, reading rate exercises, 
and in-class activities designed to increase learner motivation to 
read. An instructor’s manual of several hundred pages outlined 
course objectives and suggested materials and activities. Instructors 
were provided by the local school system, after screening by union 
and management representatives. Those who were retained to 
teach the v.ourse were able to demonstrate to these representatives 
that they could structure their teaching to meet course objectives 
and received high instmctor ratings from learners. 

The GED course involved a good deal of individualized study 
directed toward passing regularly scheduled GED tests. Learners 
used published test preparation materials as well as traditional 
school materials and workbook exercises from an extensive in-plant 
library. Use of individual learner folders, seat-work, some full-class 
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discussion, and regular individual feed-back from experienced 
GED instructors characterized how class time was spent. 

The ESL class was team taught by an experienced English as 
Second Language instructor and a Delco employee able to 
speak Italian (the first language of many but not most 
employees in the class). Activities followed exercises m several 
published ESL materials available in the Delco training center. 
Class time included teacher demonstrations in how to do 
language exercises, seat- work with both iastructors providing 
individual feedback, and full-group discussion of correct 
answers and why answers are correct. 

The TABE was used by Delco to screen learners for 
placement in the Technical Preparation class and to provide 
some diagnostic information to instructors. Demographic data 
on the class reyealed that most students were in their late 20s, 
averaged more than 12 years of education, and scored near the 
top in the reading portion of the TABE (between llth-12th 
grade levels in ability) before entering the class. 

A control group of employees not yet enrolled in the 
Technical Preparation course was interviewed and tested. 
Analysis of demographic data revealed the control group to be 
slightly older than the class group, with more males and more 
years of plant experience. In most other ways, the two groups 
were similar, however. No significant differences were found for 
education levels or for reading comprehension scores on a 
Cloze test. 



Demographic Information of Tech Prep and Control 
Groups 

Tech Prep Control 



Characteristic 
Age (mean in years)* 

Sex (M ; F)* 

Service (mean in years)* 
Education (mean in years) 
Cloze Test Scores 



27.9 


34.6 


6:8 


11 :1 


5.9 


10.8 


12.28 


12.33 


10.86 


9.58 



• Significantly different at the p< 0.05 level of 
significance 
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Subiects and LocaHons: Cumberland 



/ 



Site #2, Cumberland Hardwoods of Sparta, Tennessee, is a non- 
unionized, rural wood processing plant with approximately 300 
employees. The plant produces several hardwood products for the 
furniture industry, including drawer parts and components for 
kitchen cabinets. New technology and an ambitious quality 
assurance program have changed the nature of the work 
environment and of many traditional jobs. Cumberland has several 
small, on-going training programs. Employees in the classes 
participating in this study were all from the plant floor. 

One course at this site was entitled Communication and 
Collaboration — designed to train teams of employees involved in a 
given phase of the firm's operation. Several teams had already 
completed training in communications skills needed to work 
cooperatively as self-directed teams. The pilot study involved 
assessing two learning teams, each of 10-12 members, which the 
plant CEO described as the most difficult group of learners 
attempted so far. 

A second program at this site was an on-going GED course with 
six students enrolled. The class was taught by an experienced Adult 
Basic Education instructor employed by the company. Instruction 
followed the demonstration and seat-work pattern described for the 
Delco GED course. Earlier cycles of the GED course had allowed 
nearly 20 employees to complete the GED. However, because of 
the small number of students enrolled at the time of this study, and 
the fact that not all of them could be tested, insufficient data for 
useful analysis could be obtained for this group. 

The small size of this company prevented the formation of any 
control groups for either class. Also, because Cumberland had an 
active and successiul education program for more than three years 
prior to this study, only a small fraction of employees had not yet 
passed through the small firm’s training courses. 



Instruments 



Following a literature review for instruments and techniques 
employed to evaluate previous workplace literacy programs, a 
menu was constructed of available techniques for gathering data 
related to program impacts on productivity, learner gain, and 
learner families. 



At each site, plant-gathered indices of productivity were 
surveyed and discussed until an agreed upon list could be 
developed for the site. In addition, supervisors participated in 
developing anchored rating ^^es on information processing tasks 
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which were plant-specific (e.g., participation in meetings, doing 
quality assurance paperwork, etc.). These rating scales were 
used to rate learners before and after training. (See Appendix G 
for samples of these rating scales.) 

Interview, test, and classroom questionnaire data were 
collected for each learner before and after each course, or at 
suitable intervals for on-going classes. Lytle’s conceptual 
framework for changes in adult literacy, i.e., beliefs, practices, 
process and plans (see Chapter 3), was used as an organizing 
principle for the interview and questionnaire. Information was 
gathered on learners’ beliefs about literacy in general and their 
own literacy in particular. In addition, interviews and 
questionnaires focused upon literacy practices, the literacy 
processes and abilities demonstrated with workplace literacy 
tasks, and learners" plans for one, five, and ten years in the 
future. 

The instmments developed for the first phase involved a 
mixture of interview and questionnaire items which were to be 
used for all learners at all sites and custom-designed tasks or 
job scenarios appropriate for particular sites and classes. For the 
practices section of the questionnaire, site personnel added 
plant-specific items to a more general list of reading material 
which learners were to rate for difficulty. Questions related to 
literacy practices in work teams and in the plant were worded 
to reflect local language use. Questionnaire and focus group 
questions reflecting literacy practices with family members were 
also worded to reflect local use. For the process section of the 
model, personnel at each site participated in analyzing 
workplace literacy tasks and constructing Cloze tests and job 
scenario literacy tasks (e.g., reading plant newspapers or using 
job aids, forms, graphs, etc.) related to that workplace. (See 
Appendices A-D for .'.ample instruments and Chapters 3-5 for 
the research rationale I’cr constmction of these instmments.) 

A significant amount of instmment development occurred at 
the Delco site. Considerable time was saved at the Cumberland 
site by using the Delco instmments as models for modification 
or to stimulate the thoughts of plant personnel about what 
might be useful tasks for the custom-designed portion of the 
assessment. 

/nferwW 

An interview protocol was devised to cover all four aspects 
of Lytle’s model — beliefs, practices, process, and plans. For 
beliefs, learners were asked to describe a literate person they 
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knew at work and elsewhere as well as how literate they saw 
themselves, both now and in the future. Concerning literacy 
practices, learners were asked what reading and writing they had 
done recently, both at work and away from work. Literacy process 
was tested using three different job-related items (i.e., a newsletter 
article, a graph, and a job aid) which were selected with the advice 
of the site coordinator. The subjects were asked to describe how 
they read or used the items as well as to answer questions about 
the specific contents. Finally, learners were asked about their plans 
for one, five and ten years in the future and how they saw reading 
and education as part of those plans. 

Questionnaire and Cloze Test 

A written questionnaire was also administered to participants 
during one of the first class meetings and again nea. the end of the 
course. Items dealt with the areas of literacy beliefs and practices, 
included a Cloze test based upon the local plant newspaper, and in 
addition contained questions about family literacy for those learners 
with children between the ages of 3 and 17. To complement the 
beliefs questions in the interview, the learners were asked to write 
down four or five words that described them as a reader and a 
writer and to do the same for someone they saw as good at reading 
and writing. Further information about practices was sought 
through a checklist of 20 possible types of reading material (e.g., 
books, signs, training manuals, pay stubs, charts, cartoons); subjects 
were asked to identify the items that they had read recently and to 
rate them, on a scale of 1-5 in terms of tlie difficulty they had in 
reading them. They were also questioned about the frequency of 
literacy-related activities: how often, for example, they read a 
newspaper, made a shopping list, or visited a library, as well as 
how many books they owned. In relation to literacy at work, they 
were asked to rate on a scale of 1 (“very like me”) to 5 (“very 
unlike me”) 10 statements, such as “I just listen in meetings," “My 
ideas are discussed in meetings,” “I read information when it is 
handed out,” and “I have trouble reading information sent out by 
management.” 

The questions about family literacy concentrated on literacy 
practices, particularly frequency of literacy activities, e.g., how often 
the participant’s child looked at books, read or asked to be read to, 
or visited a library; how often the participant read to the child or 
helped with reading; how many books the parent or child bought 
in the last year. 

For each site, the coordinator helped to select a suitable passage 
from workplace materials for use in a Cloze Test, in which every 
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fifth word was left blank. These passages were of a page in 
length, with about 25 blanks to be filled in. 

Family Uleracy Focus Group Interview 

At the Cumberland site, a group of learners with children 
were interviewed about literacy beliefs and practices in the 
home. They were asked, for example, why they thought some 
children did better at school than others and what kinds of 
literacy-related materials they had available for their children. 
Questions used in the focus group interviews reflected 
categories developed by Fitzgerald et al. (1991) in assessing 
. home and parental factors related to children's success in 
school. 

ESL Checklist 

Evaluation of ESL proficiency is not easily done with paper 
and pencil measures since speaking, listening, reading, and 
writing are all involved. Typically, teacher checklists of a wide 
variety of behaviors serve as a diagnostic record and 
instructional guide and as an informal assessment of progress. 

Bronstein (1991) developed an extensive workplace-specific 
ESL checklist entitled. Benchmarks and Student Learning Profile 
for the Workplace ESL Program. Instructors at Delco reviewed 
this checklist, selected items appropriate to tiieir site, modified 
other items, and added a few items specific to their classes and 
workplace. This resulted in a list of competencies at three levels 
(beginner, intermediate, advanced) dealing with such areas as 
following instructions, looking up information, and filling out 
forms. (See Appendix F for a sample of this modified checklist.) 

Class Observation Sheet 

Classroom observations were performed by research 
personnel and on-site coordinators using a class observation 
form developed by Mikulecky (1990) and utilized by Mikulecky 
and Philippi (1990) and Philippi (1991) in school and workplace 
settings (see below and in Appendix E). The form requires 
observers to describe instructor activities and student activities 
and to make comments about the nature of class activities on a 
timed basis. Notations are then shared with the instructor to 
corroborate the accuracy of what has been observed and to 
make note of purposes for some activities. 
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Make note of time spent students actual^ reading or doing things. Also note 

time learners spend listening to the instructor. When learners are in small 
groups or worldng indtvldus^ should be mentioned. Special note should be 
made when the instructor or a student demonstrates how to do something. 



Productivity Inhrmation 

Information on productivity needed to be custom selected for 
each worksite, though there was a small degree of overlap (i.e., 
attendance and safety records). In addition, each site participated in 
constructing plant-specific supervisor ratings. 

Plant-Gathered Productivity Indicators. Management at Delco Chassis 
routinely gathered a significant amount of employee data related to 
achieving corporate goals. Researchers, working with management 
and union personnel, reviewed this data to select productivity 
indicators which could possibly be influenced by successful 
learning experiences in the workplace literacy program. Learner 
and control group pre- and post-data was collected on absenteeism, 
suggestions submitted, suggestions approved for awards, grievances 
submitted, discipline records, and workplace safety records. 

Supervisor Ratings. Extensive interviews were conducted with 
supervisors and workers to determine aspects of jobs that 
contributed to productivity and were related to communication, 
teamwork, and paperwork skills. Ten aspects of job performance 
emerged from interview data at the Delco plant and ten aspects 
were also used at the Cumberland plant. Supervisors then provided 
examples of behaviors which separated top from middle from 
bottom performers on each scale. These behaviors were used to 
develop anchored rating scales for each of the productivity 
categories. Supervisors then rated each worker on these scales both 
before and after training. (See Appendix G for samples of these 
rating scales.) 
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Data Gathering Procedures 

Procedures for data gathering varied from instrument to 
instrument. Some were written directly by the learner or 
indirectly from the learner’s comments and others by the 
learner’s teacher or supervisor or by the researcher. 

The interview and job scenarios were conducted by a 
researcher one-on-one with a learner. The researcher asked 
each question and made notes on the learner’s responses, 
pausing long enough to obtain a considered answer and using 
standard non-directive prompts and probes to elicit a more 
extensive response. The time taken for each individual 
interview was in the range 20-30 minutes. The Family Literacy 
Focus Group interview was conducted in a similar fashion and 
took about 10-15 minutes. 

The questionnaire and Cloze test were administered by the 
teacher during the class period. Each learner filled out the 
answers individually, with the teacher available to explain or 
clarify items when the learner was unsure what to do. 

The ESL checklist was completed by the teacher of each 
student in an ESL class, and the class observation sheet was 
completed by a researcher while a class was in progress. 

In some cases, supervisor productivity ratings were 
completed in conjunction with a researcher. At other times, 
rating forms needed to be left with supervisors. This divergence 
in procedure may have contributed to difficulties at the Delco 
plant in obtaining consistent supervisor ratings. (See Chapter 7 
for more details on this part of the study.) 

Data Analysis Techniques 

Data analysis was performed in two ways. Cloze test scores 
and quantifiable questionnaire and interview responses were 
recorded and analyzed statistically. Responses to open-ended 
interview questions were recorded, and then methods of 
analysis were developed to fit the nature of the responses. 

For some open-ended interview questions, categories of 
responses were allowed to emerge from data. These categories 
were then used to label subject comments. When category 
refinement allowed for acceptable levels of inter-rater 
agreement (90% or higher), category responses were recorded 
and statistically analyzed. For other open-ended interview 
questions, a holistic comparison was made between pre-test 
and post-test responses, and changes were rated as positive, 
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neutral, or negative. As with the category schemes, the criteria for 
assessing this change emerged from data, and the application of the 
scheme was subject to the same levels of acceptable inter-rater 
agreement. 

Both category and holistic ratings arose in connection with the 
interview question: 

How literate do you consider yourself to be? What makes you think so? 

Responses to this open-ended question nearly always included 
some kind of spontaneous self-rating response, using words such 
as “average,” “very literate,” “below average,” “poor.” These 
responses were categorized from lowest to highest on a scale of 1- 
5 to produce a score for each self-rating. In addition, a holistic 
rating was applied to the full response, in which change from pre- 
test to post-test was judged as positive, neutral, or negative 
according to the subject’s reported self-image and the reasons given 
for it. 

For any of the responses which resulted in numerical scores, 
statistical tests were applied to the set of scores for each group of 
subjects. Pre- and post-assessments were compared for the 
individuals in a class using a paired-sample t-test to detect gains 
brought about by the program. Where a class had a control group, 
the changes for the two groups were compared using a two-sample 
t-test. In addition, for the holistic change scores, the allocation of 
values +1, 0, and -1 to positive, neutral, and negative allowed the 
use of a one-sample t-test to find if the changes were significantly 
different from 0. In all cases, as the tests were of “no difference" 
versus “improvement," the statistical tests were one-tailed. 

Conclusion 

This study’s objective was to develop an evaluation model that 
can be used with most workplace literacy programs. A pilot 
evaluation was conducted at two very different workplaces where 
data was obtained on productivity, learner literacy attributes, and 
learners’ families. This data was gathered before and after each 
course using learner interviews and questionnaires, company 
records, and supervisor ratings of employees. Analysis of the data 
included coding, scoring, and categorizing items, and applying 
statistical tests to detect improvements that had taken place during 
the time learners were in class. 
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Chapter 7 



Results of Current Study 

Overview 

The main purpose of this chapter is to indicate which of the 
evaluation techniques employed have been most successful in 
detecting pre/post program changes in learners and their families 
and in employer objectives. This will be illustrated using examples 
from: 



• The Technical Preparation class at Delco (often 
contrasting it with its control group) 

• The GED class at Delco 

• The ESL class at Delco (making some comparisons 
with its small control group) 

• The Communications and Collaboration class at 
Cumberland (to a lesser extent, as less data 
gathering could be done there). 

Pre-test and post-test results were compared statistically and 
analytically for each class studied on each aspect of measurement 
used: learner beliefs, practices, processes, and plans; family literacy; 
and employer objectives. Program impact on learners in the Delco 
and Cumberland classes is summarized below. (Tabular data are 
available in Appendix H.) These results provide the principal basis 
for revising some aspects of the evaluation instruments and 
retaining others, as described in Chapter 8. 

Learner Literacy 

Changes in Belief 

• View of a literate person — no change 

• View of self as literate person — ^significant gain for 
Technical Preparation but not for control 

Changes in Practices 

• Reading and writing at work — significant gain for 
ESL but not for control 

• Participation in meetings — significant gain for 
Technical Preparation but not for control 

• Asking questions at work — significant gain for ESL 
but not for control, and significant gain for 
Cumberland 
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• Reading and writing away from work — significant 
gain for Technical Preparation but not for control 

• Range of reading— significant gain for GED and ESL but 
not for ESL control 

Changes in Reported Reading Process and in Reading Comprehension 

• Job-related Cloze test — significant gain for 
Technical Preparation but not for control 

• Prose reading process — significant gain in 
responses (particularly topics mentioned) for GED 
and ESL but not for ESL control 

• Job scenario questions — significant gains for all 
Delco classes on questions of various difficulties 

• Use of job aids — ^significant gain for ESL but not 
for control 

Changes in Plans 

• Plans for 1 and 5 years — ^significant gain in focus 
and literacy goals for Technical Preparation but 
not for control 

• Reading and education in plans — significant gain 
for ESL but not for control 

Family Uteracy 

• No change for GED and ESL; number of parents 
in other groups too small for statistical analysis 

Employer Objectives 

• Attendance — no significant change for Delco 
groups 

• Safety, suggestions, etc. — numbers too small for 
statistical analysis 

• Supervisor ratings — significant gain for 
Cumberland 

Changes in Learner Literacy 

Learner beliefs about their own literacy and about what it 
means to be a literate person were assessed with both 
questionnaire items and open-ended interview questions. 

Be/iefs 

Subjects’ views of what constitutes a literate person did not 
change significantly from pre-test to post-test, but it was 
noteworthy that their comments ranged quite widely beyond the 



area of reading and writing to include mention of broad-based 
intellectual and social qualities and the sorts of things literate 
persons were able to do. Descriptions of a literate person included 
such attributes as “college education,” “knows a lot,” “experienced,” 
and “has a better job,” and such abilities as “well-organized," 
“competent,” “helpful,” “concerned,” and “good at solving 
problems.” There were also the expected comments such as “reads 
all the time,” “understand what they read,” and “writes well.” 

In response to the interview question, “How literate do you 
consider yourself to be?” the Technical Preparation group showed a 
statistically significant improvement, and the ESL group also 
showed some numerical improvement. This was measured in two 
ways. Responses to this open-ended question nearly always 
included some kind of spontaneous self-rating, such as “poor,” 
“average,” and “very literate” which was scored on a scale of 1-5. 
(These three examples would score a 1, 3, and 5, respectively.) A 
holistic rating was also applied to the full response, and change 
from pre-test to post-test was judged by the reported self-image and 
the reasons given for it. These changes were rated as negative, 
zero, or positive. For example, one individual’s responses that 
received a positive rating were: 

Pre: “Not very literate — not much 

education.” 

Post: “I’m average. I’m not stupid. I have 
common sense and can read and write.” 

Another made gains at an apparently different level: 



Pre: “Fair or average — a bit above. I 

understand some words, but others I 
don’t. I’m not sure if it’s literacy or 
memory.” 

Post: “I’m more literate than I was before 
class — I understand more. I’m getting 
more interested in fiction, and fact. I 
look up words in the dictionary and 
thesaurus.” 

Pre/post changes for the self-rating and the holistic scores were 
statistically significant at the p<.02 and p<.01 levels for the 
Technical Preparation group. Control group scores showed no 
significant change. 
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A question about literacy beliefs corresponding to that in the 
interview was included in the written questionnaire. It asked the 
subjects to write down several words that described themselves as 
a reader and writer. Here, pre/post variation was due more to the 
number of words written down than to any change in the nature 
of the response. This illustrates an advantage that the interview 
had over the questionnaire in gathering richer data — a difference 
to which we shall return in later sections. 

Thus it appears that access to learners’ beliefs about literacy is 
more easily obtained through interviews than questionnaires, and 
their beliefs about themselves are more likely to change than their 
beliefs about others. It is probably most useful to see the opening 
question on the interview protocol in Appendix A (“Describe a 
person you know who is good at reading and writing”) more as a 
warm-up question to start the learner thinking about literacy than 
as one likely to produce evidence of change brought about by a 
program. The later questions, which ask subjects how good they 
consider themselves to be at reading and writing now and how 
good tfiey are likely to become in the future, appear more 
sensitive to change in pre- and post-assessments and provide a 
useful measure of the effect of a program on a learner's beliefs 
about literacy. 

PeacHces 

Learners were questioned about their literacy practices, both at 
work and at home. Concerning work-related activities, they were 
asked in the interview to describe the kinds of reading and writing 
that their work had involved during the past week, and in the 
questionnaire to rate on a scale from 1 (“very like me”) to 5 (“very 
unlike me”) a number of statements relating to contributions in 
meetings and the reading of work-related materials. 

The interview responses were assessed by a count of items 
mentioned and by holistic pre/post judgments of the breadth, 
frequency, and difficulty of the reading mentioned. In general, 
these measures showed pre/post gaint. For the ESL class (but not 
for its control group), the changes were statistically significant 
(p<.05 and p<.01 for the two measures). The nature of the gains in 
work-related reading activities is illustrated by these sample 
responses. 



Pre: “Newspaper — during break and 

lunch." 

Post: “Read check sheets for parts, 
suggestions, bulletins, QUILS, monthly 
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quality paper. Writing: Check off on 
sheet." 

Pre: “Nothing really— just put parts on the 

line." 

Post: “Bulletin at work. I can really read it 
now. The information is important. I 
read the magazine at work, also; it’s 
new.” 

The one exception to this pattern of pre/post gains for 
workplace reading was the Technical Preparation group, which was 
in class full-time and therefore had not been doing normal work for 
the duration of the course. For such full-time classes, it would be 
better to conduct the post-interviews a few weeks after the subjects 
return to normal work in order to register any changes in work- 
related reading behavior resulting from the training. 

Learner self-ratings on the statements about meetings and work- 
related reading showed very MtUe pre/post change overall, but a 
few aspects are noteworthy. For the Technical Preparation group, 
two items showed significant increase (p<.05 for both): talking in 
meetings and having one’s ideas discussed in meetings. For the ESL 
group, but not its control, the following item showed a significant 
increase Qx.OSy. 



When you need to know soniefhing at work, you usually ask 
someone about it. 

very like me 1 2 3 4 5 very unlike me 

This result reflects the emphasis on oral work in the ESL class 
and shows a gain in confidence by the workers, 'fhe Cumberland 
Communications and Collaboration class, which put much 
emphasis on working cooperatively with other workers, also 
showed significant gains (/K.02) on this item. In both cases,’ skUls 
dealt with in class produced changes in workplace behavior. 

Turning now to literacy activities away from work, in the 
intemew the learners were asked to describe the reading and 
writing that they did away from work, and tfie questionnaire asked 
them to rate themselves on the frequency of their involvement in 
several literacy-related activities and their ownership of reading 
materials, 

The interview responses were assessed by a count of items 
mentioned and by holistic pre/post change judged by the breadth, 
frequency, and difficulty of the reading mentioned (as described 
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above for workplace reading). The Technical Preparation class 
showed statistically significant increases in both the count of items 
and the holistic rating (p<.02 and p<.005 respectively); the control 
group showed no such increases. The GED group also registered 
some numerical gains, but the ESL group did not. The lack of 
change in reading and writing for the ESL class may be due to the 
emphasis on oral work already mentioned or to their use of 
English at work but their native language outside the workplace. 

For the section of the questionnaire about frequency of 
literacy-related activities away from work and ownership of 
reading materials, no items showed significant change. It may be 
that changes in these areas of behavior and ownership are slow to 
take effect, requiring more than the few weeks of time available 
between pre- and post-testing in this study. In their post- 
interviews, a number of the learners expressed their positive 
intentions in such areas, but the stimulus of the course was then 
only beginning to produce changes in behavior. Comments of this 
kind included: 



“I have to do more reading for my 
daughter — especially now that I have more 
incentive from this class. It’s like a spark.” 

“After this course. I’m going back to night 
school. I’m really impressed with this class.” 

Though tiie Technical Preparation class did not improve 
significantly in home literacy behaviors, their improvement scores 
were significantly better than those of the control group (p<.01), 
which actually reported less home reading in the post-test. 
Perhaps this reflects a baseline of less general reading in summer 
(when the post-tests were conducted) because of other leisure 
activities. 

In the questionnaire, learners were also presented with a list of 
20 types of reading — some general (e.g., newspapers, books, bills) 
and some plant-specific (e.g., Delco Doings, suggestion forms, 
route sheets, paycheck stubs). They were asked to rate each on a 
scale from 1 (easy) to 5 (hard) and to indicate which ones they 
had read in the last month. The results revealed a statistically 
significant wider range of reading in the post-assessment for the 
GED and ESL groups (p<.01 and p<.002). The ESL control group 
did not show such gains. 

Few of the individual types of reading showed significant 
change, but over half (about 11 of the 20 items averaged over the 
four classes) were rated by learners to have greater perceived 
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difficulty in the post-assessment. This may mean that learners were 
being more realistic after greater exposure to reading generally, or 
just that they were unable to apply the scale consistently over the 
time gap between pre- and post-test. For those not accustomed to 
using scoring schemes, there may be a problem in such assessment, 
particularly self-assessment. (See also supervisor ratings in the 
Productivity section.). The wording of the question may also have 
contributed to the difficulty; if the learners had been asked first to 
indicate which of the types they had read recently and then to rate 
only those items, it is possible that more consistency might have 
been obtained. 

These difficulties point up once again the inflexibility of a 
questionnaire and its dependence on the ability and willingness of 
the person filling it out, compared with an interview in which the 
interviewer can explain a question and probe for further 
information to clarify the learner's intentions. 



Process and Ability 

In the interview, workers were asked to respond to both 
process and content questions on a plant newspaper article, a 
moderately complex graph, and a job instruction sheet. They were 
also given a Cloze test construaed from plant reading material. 

JThe Cloze test used at Delco came from a plant newspaper 
article. The Technical Preparation class made statistically significant 
pre/post gains (/K.02), while its control group did not. The GED 
and ESL groups also did not make significant gains, but the reason 
here appeared to be that the reading passage was too difficult for 
them. These two groups had mean scores of about 7 out of a 
possible 23, which indicates a frustrational reading level, but the 
Technical Preparation class and its control group had means of 10 
or 11, well above the frustrational level. (50% replacement indicates 
an independent reading level; below 35% indicates a frustrational 
reading level.) Given this range of reading ability, it would have 
been better to have had available two (or even three) Cloze tests of 
different difficulty levels, to be used as appropriate. 

The Cumberland Cloze test used the plant safety rules, and here 
the mean score was 14 out of a possible 25, showing that the test 
was well within the reading ability of those taking it. However, no 
pre/post comparison was available at Cumberland since the on-site 
coordinator managed only a single administration of the test. 

A portion of the interview involved responding to a newspaper 
article, a graph, and an instruction sheet in job-related scenarios. 
Learners tended to answer correctly the simpler, fact-level content 
questions on the pre-test, and no gain in this area was apparent on 
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the post-test. Significant gains were demonstrated on more 
complex items, which often called for the use of inference and 
interpretation. For example, the Technical Preparation group 
showed significant gains on the most difficult article and graph 
questions (e.g., “What happened to the inventory value in August 
and September?” — ^which required learners to describe a trend), 
while the ESL group did so on those of medium difficulty (e.g., 
“What is the inventory value for the week of August 19?” — which 
required reading from two scale values). Since the levels of 
competence of the different groups (and individuals) varied, a 
greater range of difficulty in the sets of questions would have 
allowed improvements to show at an appropriate level. This 
would also have been assisted by a wider range of item types in 
each section, from the simple factual to more difficult 
interpretation questions. 

When asked to read a plant newspaper article and describe 
how they read it (i.e., the processes they were using), learners’ 
responses covered two main areas: reading strategies and topics 
of interest. Strategies included skimming, starting with headings 
and bold print, and reading the first and last paragraphs. Topics of 
interest included the products manufactured by competitor 
companies and the wages that they paid. Responses to this 
process question (reproduced below) included the following 
examples: 

Describe what you would look at. What would you be 
thinking about? How would you go about reading 
this story? 

Pre: “Check each heading and decide 

whether to go further.” 

Post: “Read the headings, get ideas about 
the companies, skim, know what they 
make, and know their customers and 
competitors.” 

Pre: “Read the first paragraph, read the 

headlines and bold print, and pick out 
what hits my attention.” 

Post: “Read the dark print first, then break it 
down from there and read what gets 
my attention. Also, I’d find out what 
it’s about and what they are telling in 
it. Then, I’d read it in depth.” 
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Pre: “Title, first paragraph, through the 

whole thing — ^analyze it.” 

Post: “Title, subject iDelco Doings), and 
function and operation of companies. I 
would look at Asia and Europe 
(competitive markets) to see how their 
prices are lower and higher.” 

Pre: “First the title, then I’d read from 

beginning to end.” 

Post: “Start at the beginning. Look at the 
areas and read all the way through. I’d 
also read about how Delco is trying to 
compete, its main customers and the 
percentage of wages and benefits. I’d 
read Delco Doings to find out about 
Delco’s further needs. Reading these 
things makes you familiar with other 
companies.” 

These responses were analyzed by counting the number of 
separate items mentioned by the interviewee. For all the Delco 
classes, the total number of responses and the number of topics 
mentioned increased numerically, and these were statistically 
significant for the GED and ESL groups (p<.005 in both cases); the 
control groups showed no gains in these areas. The increase in the 
number of topics that the learners mentioned shows a greater 
ability to make connections between what they read and their own 
knowledge, as well as a growth in confidence arising from their 
time in class. For the most part, increased discussion of strategies 
included comments one would expect from more sophisticated 
readers. 

In connection with the job instruction sheet, learners were 
asked about their use of such job aids, how long it took them to 
read one, and how difficult they found it. The only case of a 
statistically significant gain was for the ESL class (but not for its 
control group) in reply to a question on how likely they were to 
use a job aid. It appears that ESL learners’ confidence in 
approaching job-related reading had been increased by their 
attendance in a class. Other questions that involved self-reporting 
of reading skills were not successful because of the interviewees’ 
inability to gauge their own capacities. A question about the length 
of time it took to read a job aid produced a wide spectrum of 
answers: from one or two minutes up to a week. Responses at the 
bottom end of this range were clearly unrealistic; just one of the 
content questions tended to take more than two minutes for most 
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learners. Responses like a day or a week seemed to refer to the 
length of time needed not only to read the job aid but also to 
learn the job it related to. Ambiguity in connection with such 
items may well preclude their effectiveness. 

In all of the job scenarios (newspaper article, graph, and 
instruction sheet), responses to the content questions showed 
considerable variation, both between groups and between 
individuals. To accommodate this variability, the set of questions 
for any one section needs to range in difficulty and in nature 
(involving fact, inference, and application), so that there is room 
for improvement from pre- to post-test at some level for all tliose 
interviewed. 

Plans 

In the interview, learners were asked about their plans for the 
future 1, 5, and 10 years ahead, and to explain how reading and 
education formed part of those plans. Assessed on how definite 
and detailed the plans were, the Technical Preparation students 
showed significant pre/post improvements for one- and five-year 
plans (p<.02 and p<.05); this did not occur with the control group. 
The ESL class showed a significant increase in references to 
reading and education as part of their plans (p<.005), a result 
which was not repeated in its control group. 

Responses to planning questions ranged from mentioning 
prospects for advancement, in the company or out of it, to 
intentions regarding marriage, children, housing, and retirement. 
The following are typical responses to the question; 

Wbat are your plans for the next year? 

Pre; “Finish degree.” 

Post; “Getting married. Going to school — 
four nights a week in the fall.” 

Pre; “Have another child, learn new jobs in 
the same department, and get a new 
car.” 

Post; “Have another child, lose weight, take 
some course (I don’t know what 
kind), and probably finish a degree in 
retail.” 

What are your plans for the next five years? 

Pre; “Apprenticeship completed.” 

Post; "Have electrical apprenticeship, have 
kids — maybe, (and) perhaps buy 
another house.” 
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Pre: “Go through the apprenticeship. This 

will take up a big amount of time.” 

Post: “Be done with the apprenticeship and 
definitely move up and off the 
assembly line.” 

Pre: “Ending an apprenticeship — 

journeyman or greater position.” 

Post: “I’d like to have a combination of 
school and owning my business.” 

The connection with literacy was made more explicit through 
the follow-up question about the role of reading and education in 
their plans, as these comments from the post-interview reveal in 
response to the prompt: 



Explain how you see reading and education in these plans. 



“Reading helps with eveiything. As you grow 
you learn. I want my life to grow.” \ 

“I need to develop and build confidence — do 
more reading — that’s definitely important. Get 
my kids to do more of it.” 

“Get a better job by taking more classes. Help 
my kids read more — help them in school.” 

“If you can’t read, you can’t troubleshoot the 
machines.” 

“The more I learn, the easier it is to make 
suggestions about things and to apply for 
better positions.” 

“I feel better after being in here and I want to 
learn more. I have to read a book on game 
for hunting. To retire, I need to read about 
benefits.” 

Overall, the learners were very positive about their experiences 
in classes and saw them as opening new doors, both for further 
education and for a life of greater opportunity. 

Changes in Family Literacy 

Measures of family literacy mainly involved information about 
how parents interacted with their children in literacy-related 



activities. In addition, some of the parents’ reading behavior away 
from work reflected upon changes in family literacy. 

Learners with children between the ages of 3 and 17 were 
asked in the questionnaire about the frequency of such activities 
as reading to their child, helping the child with reading, and 
buying books for the child. Also, they were asked how often the 
child read alone, and what kinds of books or other materials (if 
any) the child borrowed from a library. 

These questions revealed no statistically significant changes for 
the GED and ESL groups, the only groups in which the number of 
parents was large enough to draw any conclusions. These groups 
each contained 12 parents of children in the relevant age range, 
while the Technical Preparation class and the Cumberland group 
contained only 4 each. 

However, the responses to the questions on family literacy did 
show slight overall gains, even though not statisticaUy significant 
ones. It may be that the time between pre- and post-tests was not 
enough for any changes brought about by the classes to have 
much effect. It may also be that a larger sample size would have 
revealed the trend for improvement to be statistically significant. 
In addition, there is some evidence about reading practices in the 
interviews to indicate a movement towards more literacy activity 
on the part of the learners and their children. During post-test 
interviews, class members were more likely to report newspaper, 
magazine, and novel reading. Two parents who had not 
previously mentioned reading to children, mentioned “reading to 
my^child” or “reading a children’s book to my son” in their post- 
interviews. Another reported reading child care books and 
magazines on parenting. These had not been mentioned in the 
pre-interviews. One class member commented, “I definitely read a 
lot more since I started taking this course.” 

Changes in Meeting Employer Objectives 

In relation to worker productivity, measures used at Delco 
were attendance, safety records, suggestions submitted, 
suggestions approved for awards, grievances submitted, and 
discipline records. In addition, each site participated in 
constmcting plant-specific supervisor ratings. 

No significant changes occurred in learner attendance, but 
because of the small sample size the absences of a few individuals 
could affect the total quite markedly. For example, in the 
Technical Preparation group, half the absences in the post-training 
period were attributable to three employees. With samples this 
small, extreme caution should be used. 




7 '/ 



76 



TR93-2 



Other measures such as productivity suggestions and accident 
records involved numbers too small for statistical testing. 
Suggestions were made by only a few members of each group 
during the periods concerned, and these followed no apparent 
pattern. Accidents were even rarer; for example, no Technical 
Preparation or control group member had an accident during the 
six weeks prior to training. Such figures do not allow statistical 
analysis. 

At both Delco and Cumberland, supervisor ratings were devised 
to measure aspects of jobs that contributed to productivity and 
were related to communication, teamwork, and paperwork skills. 
Extensive interviews were conducted to determine relevant skills 
and, at each site, ten aspects of job performance emerged from 
interview data. Supervisors then provided examples of behaviors 
which separated top from middle from bottom performers on each 
scale. These behaviors were used to develop anchored rating scales 
for each of the productivity categories. Supervisors then rated each 
worker on these scales both before and after training. 

At Delco, the supervisor ratings of the workers’ job-related skills 
produced some anomalies that cast doubt on the consistency of . the 
ratings from pre-test to post-test. Even though supervisors 
participated in the scale development, some seemed to rate some 
workers exactly the same on all scales. Some of the ratings 
appeared to be carelessly done. Even with certain items and 
individuals removed to correct for this, no change was apparent. It 
, may be that supervisors need more training or instruction before 
doing ratings, or that a time period of six weeks may be too short 
to register improvements. 

However, all ten aspects of the assessment scheme used at 
Cumberland showed significant improvements (p<.0001) over the 
11 weeks of the classes. Here, just two individuals made the 
assessments and made them for the same workers in both pre- and 
post-tests, whereas at Delco up to four supervisors assessed the 
members of each group and there had been some personnel 
changes between the pre- and post-tests. Also, the Cumberland 
assessors had slightly more education and were not shop-floor 
supervisors, as at Delco; they may have had more experience in 
making judgments and ratings. 

Another factor that may have contributed to the Cumberland 
results is the choice of assessment categories. These were very 
closely related to the objectives of the Communications and 
Collaboration course, covering such items as communication skills, 
problem-solving ability, and conflict resolution. The Delco 
assessment referred mainly to specific job skills such as machine 
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setting and record-keeping, but Delco courses were of a more 
diffuse job training nature, not relating directly to these skills. This 
tends to confirm the notion that learners gain knowledge and 
skills only in the areas that are taught. 

Conclusions 

The instruments used in this study to measure the impact of 
literacy programs varied in their success at detecting pre/post 
changes in learner literacy, family literacy, and employer 
objectives. The main results and some observations about 
assessment utility follow. 

Learner Literacy 

Beliefs 

• Learners reported improvements in their view of 
themselves as literate, but not in their view of a 
literate person. 

• Interview questions were more successful than 
questionnaire items in detecting changes in self- 
image. 

PracHces 

• Reading practices at work improved in areas that 
related to the class attended. 

• Full-time classes need to be post-tested some time 
after learners return to normal work, so that 
changes in work-related reading can take effect. 

• Reading practices away from work improved for 
classes where home reading had been 
encouraged. 

• Interviews were more sensitive to changes in 
reading practices than were questionnaires. 

• Self-assessment of reading difficulty produced 
some inconsistencies that cast doubt on this 
questionnaire section. 

Process and Ability 

• Cloze test scores improved only when the 
passage was at an appropriate reading level, 
suggesting a need for several different test 
passages. 
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• Answers to process questions on job-related 
reading materials showed improvements in reading 
strategies and topic connections. 

• Answers to content questions on job-related 
reading materials showed improvements at various 
levels for different classes, suggesting a need for a 
range of difficulty and type in the questions. 

Plans 

• Learners were generally more definite and detailed 
in their plans for the future after attending classes. 

Family Uteracy 

• Questionnaire items showed slight gains in some 
areas, but the time may have been too short for 
significant improvements to occur. 

Employer Objectives 

• Attendance showed no significant changes; with 
small groups, there is a problem of a few 
individuals’ absence distorting totals. 

• Safety, suggestions, etc were too infrequent for analysis. 

• Supervisor ratings showed significant gains when 
areas covered related to the class as well as the 
job. 

• Consistent supervisor ratings are difficult to obtain 
across several supervisors and when personnel 
change from pre- to post-test. Education and 
experience levels of supervisors may also be a 
factor in obtaining consistency. 
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Chapter 8 



Discussion and Impucations 

Overview 

This pilot study has shown that it is possible to perform a 
broad-scale assessment of workplace literacy programs in order to 
measure the impact on learners, their families, and their 
productivity. The results of the study demonstrate some 
improvement in each aspect of the assessment model. However, 
gains appear to be limited to what is taught; there is very little 
transfer to areas not addressed by instruction. 

Learner change was measured in the areas of beliefs, practices, 
processes, and plans. Where these formed a part of class 
instruction, learners made gains in the following areas: 

• Their literacy self-image 

• Their ability to articulate plans 

• The ampunt and range of literacy activity both at 
work and away from work 

• Reading strategies and comprehension 

Classes did not address directly issues of family literacy, and 
little change was evident in this area. Productivity measures proved, 
on the whole, to be unsatisfactory for the small numbers of learners 
studied, although supervisor ratings showed increases when areas 
assessed were closely related to instruction and company goals. 

The evgluation model itself was also under scrutiny in this 
project. Several points of interest have arisen from the pilot 
assessment. 

• Questionnaires, although time-efficient, seem to be 
less effective than interviews in gathering accurate 
information. 

• Because of the range of learner abilities, workplace 
scenarios need to include questions at a variety of 
difficulties; Cloze tests of varying difficulty may 
also be necessary. 

• It is desirable to have direct measures of learner 
productivity as well as more reliable ways of 
obtaining supervisor ratings. 



NATIONAL CENTER ON ADULT LITERACY 



81 



81 



1 



Principal Achievements 

A good deal has been learned from this pilot assessment. The 
pilot study has demonstrated that it is possible to perform a broad- 
scale assessment of workplace literacy programs using learner 
interviews, tests, and questionnaires in reasonable time-frames 
(i.e., 20-30 minutes for interviews, 10-15 minutes for tests and 
questionnaires, before and after instruction), as well as company 
records and supervisor rating scales. During subsequent studies, it 
will be determined if the assessment model can be transferred to 
additional workplace literacy programs with a minimum of 
technical assistance. 

Secondarily, results from the assessment provide indications of 
what effective workplace literacy programs can accomplish and 
may not be able to accomplish. Discussion of tfiese results will 
reflect and substantiate two major generalizations: 

• Workplace literacy program instruction is able to 
demonstrate positive improvement in eacli area of 
the assessment model, i.e., beliefs, practices, 
processes and abilities, plans, productivity, a»*d 
family literacy. 

• Gains seem to be limited to areas directly 
addressed by instruction, i.e., programs and 
classes accomplished gains only in areas where 
there was direct instructional activity. No clear 
carry-over or transfer to other areas is apparent in 
evaluation results. 

Impact of Programs and Link to Instruction 

Data were gathered for the learners in a range of classes: 
Technical Preparation, GED, ESL, and Communication and 
Collaboration. Program impact results will be summarized and 
discussed in direct relation to the types of instruction in classes 
where gains were made. The types of instruction in classes not 
demonstrating gains in particular areas of the assessment model 
will also be discussed for comparison purposes. 

fie/iefs Ahouf Self as Uterafe 

Changing adult learners’ beliefs about their own literacy 
abilities is important for several reasons. Adults with a negative 
impression of their own abilities are not likely to attempt literacy 
away from the supportive environment of a classroom and 
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nurturing instructor. Significant growth in literacy abilities requires 
hundreds of hours of practice — ^more than most programs can ever 
provide in class time. For this reason, it is important that learners 
become more independent by developing more positive self-beliefs 
about their literacy abilities. They need to see themselves as 
capable of attempting more literacy and practicing more on their 
own, as opposed to avoiding literacy tasks and literacy practice. 
(Incidentally, it is important that learner beliefs about their own 
literacy be accurate or they will feel betrayed if they discover they 
have been wrong, and that betrayal can lead to abandoning 
altogether much of what has been learned in classes). 

Data about learner literacy beliefs were collected in all classes. 
Except in the GED class, learners demonstrated improved views of 
themselves as literates. This was mainly revealed during interviews 
through more positive self-descriptions and self-assessments. 

In The Technical Preparation class, learners were able to monitor 
their own progress on reading comprehension and reading rate 
through class tests and discussions. In addition, class discussion 
time during seven-hour learning days often addressed future 
learning plans and why the skills studehts were mastering would be 
of use in future training, 

The ESL class, similarly, used class discussion both to provide 
English practice and to highlight the relevance of what was being 
learned to future use. Learners were asked to share, in journals and 
later oral discussion, personal accomplishments in written and oral 
English. This activity served both as an instructional tool to improve 
language use and as a feedback mechanism for reinforcing learners' 
views about their own growing language and literacy competence. 

The Communication and Collaboration class revolved 
substantially around the concept of joint and personal goal setting, 
planning for accomplishing goals, and monitoring effectiveness. 
Some goals related to direct job performance, but a substantial 
number related to improving individual communication abilities. 
Considerable time was expended on both individual and group 
monitoring of gains. An apparent result was expanded and 
improved beliefs on the part of learners about their own literacy 
abilities. 

Tlie GED class demonstrated no gains in improved learner 
beliefs about tJjeir own literacy abilities. The structure of classes did 
not lend itself to substantial instructor feedback, group feedback, or 
individual monitoring in this area. Most work was individualized 
and directly related to completing practice exercises for the GED 
tests. Interviews with learners often indicated a workmanlike 
attitude toward how many exercises they had gone through with 
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little sense of improved individual abilities beyond the class. 
Instruction did not focus on internalizing a sense of expanded 
persona! abilities, and assessment did not reveal such chariges in 
belief to have occurred. It is important to note that other 
assessment measures indicated that GED students were learning. 
They had actually improved in some literacy abilities and 
practices. The significant factor here is that little class time was 
directed toward identifying and reinforcing growth in this area 
and concurrently no change in individual beliefs about literacy 
abilities was demonstrated. 

Plans 

Interview questions about future plans and the relationship of 
literacy to those plans were asked of learners in the Technical 
Preparation, GED, and ESL classes. In the Tech Prep class, which 
was designed as a prelude to further training, a good deal of time 
was spent aac.essing study skills and the demands of future 
training. Post-class interviews in tliis class revealed plans which 
were articulated with more focus and detail than had been true of 
tlie pre-class interviews. A similar pattern occurred in the ESL class 
which sometimes used discussions of learners’ futures as an 
activity for improving the use of oral English. Learners in the GED 
class, who primarily focused on passing the GED test and were 
involved mainly with individual seat-work, demonstrated no 
measurable change in the clarity or focus of their plans and made 
no greater mention of education and literacy as parts of future 
plans. 

literacy Practices 

Changes in the amount and types of literacy practices used by 
learners were assessed by a combination of interview questions, 
questionnaire checklists, and rating scales. The same pattern of 
gains directly related to classroom focus areas was revealed in 
assessment results in all classes. 

The Technical Preparation classes emphasized improved 
reading habits. One instructor even took learners to the library, 
read portions of books and magazines aloud in class, and 
emphasized improved literacy habits. Even though the course title 
implied workplace applications, in post-interviews learners 
demonstrated increased reading and writing at home compared to 
the literacy behaviors of the control group. Employees from only a 
few work-teams were enrolled in these classes, and they 
participated in a significant amount of group work in class with 
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team members. Statistically significant increases in willingness to 
offer ideas and discuss them in quality assurance meetings were 
also reported by learners. 

The ESL learners spent some class time on workplace literacy 
habits and practices by reading bulletin board items, newsletters, 
and job materials in class. Gains in pre/post literacy practices at 
work reported by ESL learners were significant. These gains were 
also significantly greater than those of a small ESL control group 
which served as an indicator of changes that result simply from 
living in an English-speaking environment for a comparable period 
of time. Little class time was allocated to home literacy activities, 
and no significant improvement in this area was noted. Oral 
discussion of class exercises (such as how to convert statements to 
questions) took a significant amount of class time. The heavy 
emphasis on oral language and asking questions led to significantly 
higher ratings for asking others for information in the workplace. 

Learners in the Communication and Collaboration class also 
participated in a significant amount of group activity. They, too, 
showed gains on items involving question-asking and 
communicating in the workplace. Lack of post-assessment 
interviews prevented more extensive examination of changes in 
other literacy practices. 

The GED class, which did not emphasize literacy practices at 
home or at work, did not show gains in these areas on interview 
items. Questionnaire checklists, however, revealed a statistically 
significant tendency for learners to report attempting a wider 
variety of reading and writing. 

An interesting phenomenon was noted in GED and ESL learner 
responses on the checklist. In addition to asking what types of 
materials they had read recently, learners were asked to indicate 
the difficulty they had reading these materials. Learners in both 
classes indicated that materials were harder to read at the end of 
the classes than they had been at the beginning of the classes. This 
finding suggests that low literates without much experience with 
reading may initially over-estimate their own abilities. It furiher 
suggests that instructors should not simply propose extra reading 
without providing support. When low literates find that even simple 
materials are more difficult than they anticipated, they may become 
discouraged and retreat from reading them. 

Reading Processes and /dsiliHes 

The most psychometrically rigorous measure of reading ability 
used in this study was the Cloze reading test. The Technical 
Preparation class, which spent the most time in reading practice, 
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was the only group to demonstrate a statistically significant gain 
on this measure. GED and ESL learners, who found the newsletter 
story used in the Cloze test to be at their frustrational reading level 
when instruction began, found the high-school difficulty article still 
at frustrational reading level at the end of instruction. It is unlikely 
that learners in either group received enough reading practice to 
bring this brief story about General Motors vehicles within their 
comprehension ranges. 

Job literacy scenarios provided a more diverse range of 
indicators of learner literacy processes and abilities across several 
types of workplace materials. Questions assessing the strategies 
used by learners in reading newsletter stories, graphs, and job aids 
revealed some change in the sophistication with which learners 
read. The Tech Prep class, which was comprised of high school 
graduates and several learners with some post high school 
education, scored very high on pre-class measures of how they 
went about reading. This class spent a good deal of time 
addressing study skills and reading strategies, and learners did 
score numerically higher on post-assessments than their already 
high pre-scores. Ceiling effects here made statistically significant 
improvements difficult to attain. ^ESL and GED learners 
demonstrated especially significant gains in topics focused upon. 
Even though they had not improved in reading abilities enough to 
do well v/ith the earlier reported Cloze newsletter article, the 
reading practice received during their limited hours of instruction 
had led to a more sophisticated approach to reading. Control 
groups, who received no instruction, demonstrated no 
improvement from pre- to post-assessments of reading strategies. 

Comprehension questions of increasing difficulty were also 
asked in the different job literacy scenarios. Learner gains again 
reflected learner instruction. Technical Preparation learners, whose 
extensive class work addressed inference and problem-solving 
tasks, improved most on the more difficult scenario questions. The 
ESL learners, who met eight hours each week and spent some 
time with workplace materials, improved most on middle level 
difficulty questions. The GED group, who met only four hours per 
week and did little with workplace materials, demonstrated gains 
on only one comprehension question related to workplace 
literacy. Once more, gains appear to be directly related to the type 
and amount of instruction received by learners. 



Family Literacy 

Some impact of instruction on home literacy has already been 
discussed in an earlier section on literaq^ practices. Learners in 
classes which focused upon home materials did improve home 
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literacy practices; those in classes which did not focus on home 
materials made no changes. Only a relatively small number of 
learners had children and were therefore qualified to answer the 
family/parent literacy questionnaire items. No class spent direct 
instructional time on family literacy, and no significant gains were 
noted in this area. Though not statistically significant, a few 
Technical Preparation parents took their children to the library 
more often after their own class was taken to the library by their 
instructor. Similarly, a few parents reported reading to children 
slightly more often. These accounts were infrequent. It appears that 
benefits of instruction do not transfer very far beyond the focus of 
actual instructional activities. 

ProducHvity 

For the variety of reasons discussed in Chapters 6 and 7, few of 
the productivity assessments proved satisfactory for groups as small 
as the 12-15 member classes and control groups. There is some 
indication, however, that some productivity gains were directly 
related to type and amount of instmction received by learners. The 
only group to demonstrate consistently improved supervisor ratings 
in workplace-based communication and literacy use was the 
Cumberland Communication and Collaboration class. The entire 
class was structured to address these workplace communication 
demands. Results suggest that this focused instruction was effective 
in areas directly related to company goals. No comparable gains 
were demonstrated with Delco supervisor ratings, though difficulty 
in obtaining acceptable ratings clouds this finding. Questionnaire 
items dealing with participation in team meetings indicate that both 
Delco and Cumberland learners who participated in class group 
work and discussions did make significant gains in team meeting 
discussion participation. Again, the relationship between instruction 
and improved performance is fairly direct. 

Conclusions from Results 

In workplace literacy instmction, it appears that you get what 
you pay for and not much more. Classes and instmctors at the two 
sites demonstrated that what you choose to spend time on in class 
matters a great deal. Statistically significant gains were made by 
some students for every segment of the evaluation model. More 
detailed analysis reveals that gains occurred, however, only in areas 
directly addressed by instmction and class activity. 

This is both good news and bad news. It is heartening to know 
that instmction works. Workplace literacy programs that focus on a 
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specific goal and provide significant instruction toward that goal 
can help learners improve. If time is spent providing learners with 
feedback about their improved literacy performance and 
developing literacy habits at home and work, workers will 
improve their self-concepts about their own literacy and will read 
more. The bad news is that hopes for broad transfer from 
relatively brief programs (as nearly all workplace programs are) 
appear to be misplaced. Whatever effective class activity focuses 
upon is the major area of gain. Even improvement in literacy 
practice appears tightly related and limited to classroom practice. If 
class time focuses only on workplace activities, practices appear to 
improve only with workplace literacy materials. For productivity to 
improve, instruction needs to focus directly on activities involved 
with production. Extra dividends of transfer to improved family 
literacy seem unlikely unless instructors also spend time with 
family literacy activities. 

This implies the need for some hard decision-making by 
instructional planners. The results of this study make it much more 
difficult to accept the contention that any single focus of literacy 
instruction will bring improvement in a multitude of areas. It also 
suggests that diffuse instruction which touches lightly on many 
areas will not bring about gains of any significance in any 
particular area. The GED group improved a little in general 
reading strategies and may have improved in taking the GED test, 
but there are no indications that much more occurred. 
Instructional planning did not focus on much beyond the GED 
goal. The ESL group improved in oral activities and in some 
workplace activities for which they received instruction, but not in 
areas where they received little instruction. This same pattern, i.e., 
instruction directly and narrowly related to gains, holds for the 
Technical Preparation class and the Communication and 
Collaboration class. 

This should not be taken to mean that workplace literacy 
instruction should always focus upon a single workplace goal. It is 
likely that the most beneficial mix is instruction which expands 
learner practice time beyond the classroom by improving worker 
literacy practices and beliefs at home and the workplace. Since 50 
or even 200 hours of class time are not sufficient for many learners 
to reach their full potential, precious class time must be used, in 
part, to increase literacy practice and learner independence. If 
productivity is an issue, workplace materials and activities used in 
class should be directly related to materials and activities 
employed during production. If other goals are desirable, they 
must be planned for, and it seems likely that additional learning 
time will also be needed. 
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I What Has Been Learned About How to Evaluate Programs 

One of the major goals of this study was to develop a model for 
evaluating workplace literacy programs. For the most part, the pilot 
assessments validated the utility of a broad-based conceptual 
framework of adult literacy learning in the workplace. It was 
possible and productive to note gains in areas of learner literacy 
beliefs, practices, processes and abilities, plans, productivity, and 
family activities. A good deal was also learned about the limitations 
and pitfalls of particular evaluation approaches and methods. 

Limitations of Questionnaires 

Time is at a premium in workplace literacy programs. Many 
programs are only able to provide brief instruction, and still others 
lose money for each hour of learner time since learners are not 
producing a profit while in class. To the degree that checklists and 
questionnaires can be used to gather information, as opposed to 
individual interviews, a substantial time saving can be made. This 
pilot assessment used overlapping oral interview and written 
questionnaire items to test the degree to which the assessment 
approaches produced similar findings. For the most part, 
questionnaires, though time-efficient, were much less effective and 
accurate than even brief face-to-face interviews. This was especially 
true in the areas of literacy beliefs and practices. On written forms, 
learner responses in these areas were very brief — even from the 
I more literate Technical Preparation learners. Interviewers, however, 
waited until learners paused in speaking and then asked, “Anything 
else?” or “Can you think of any other examples?" until they received 
a “no.” This produced a good deal more information and more 
accurate representations. Questionnaire responses in these ar^s 
probably more closely reflect the degree to which learners could 
read and wanted to write. The questionnaire responses tended to 
reinforce the interview responses, but questionnaire assessment 
was often not sensitive enough to detect changes— especially on 
global questions about literacy beliefs and practices. 

Questionnaires were effective when they could be focused. For 
example, descriptions of literacy behaviors in team meetings, 
listings of recently-read materials specific to a workplace, and 
I descriptions of literacy behaviors with one's children elicited 
I information rapidly. In cases where there was an overlap between 

I interview questions and questionnaire items (e.g., home literacy 

I behaviors in the interview and family literacy behaviors in the 
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questionnaire), triangulation revealed the questionnaire items to 
reflect accurately the more extensive oral comments. 

Job'Rdated Scenarios 

The job literacy scenarios were custom-designed to reflect 
workplace literacy tasks of importance at each worksite. They 
attempted also to reflect the range of reading types present in 
national assessments of adult literacy (i.e., prose reading, 
document reading, and quantitative reading). The scenarios 
provided, as much as possible, a realistic purpose for reading and 
attempted to assess both how the learner went about reading 
(processes) and how well the learner understood and could use 
information from the reading (abilities). 

These job literacy scenarios proved to be quite productive in 
assessing improvements in the sophistication with which learners 
approached reading tasks. The initial scenarios, which were 
limited to a very few comprehension questions, were somewhat 
productive, but need to be expanded to reflect more accurately 
gains in several types of readl-ig (searching for facts, drawing 
inferences, and making applications beyond the task at hand). 
Instruments in Appendix A have been revised to reflect these 
changes. 

Cloze Tests 

The Cloze test was simple to construct and relatively easy to 
administer. Instructors reported little difficulty with the test, which 
provided a sample sentence demonstrating how to fill in blanks. 
With the Technical Preparation and Cumberland classes, the 
material selected was well within initial comprehension ranges. 
This was not true for GED and ESL learners at Delco, however. 
The story, written at a high school level of difficulty, was beyond 
most learners both before and after instruction. For low level 
learners, it would be desirable to construct a second Cloze test 
using simpler workplace materials. 

Though some instmctors at pilot sites were familiar with the 
Cloze test procedure, others were not. Directions for how to 
develop and interpret Cloze tests were created for instructors. 
These are included in Appendix C with samples of Cloze tests 
developed at the pilot sites. 
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Family literacy Questions 

Both oral focus group methods and written questionnaire items 
related to family and parent literacy were piloted for this study. 
Both seemed effective in gathering information. However, 
workplace literacy classes are small, and the number of parents at 
the pilot sites was even smaller. For most items, these small 
numbers precluded meaningful statistical analysis of workplace 
literacy impact on parent literacy practices. These measures are 
likely to be of more use for special programs which focus upon the 
workplace/family connection or for much larger groups. 

Employer-Gathered Productivity Indicators 

Though previous studies have discussed the need for assessing 
productivity impacts of workplace literacy programs, few have tried 
to do so. This pilot assessment attempted to use some of the 
indicators of productivity suggested in the research literature (i.e., 
attendance, accident reports, useful productivity suggestions made 
by employees, etc.). The pilot test revealed that it is possible to 
gather such data with a minimum of effort on the part of 
employers. It also revealed that the information is not of great use if 
sample sizes are small and time between assessments is not very 
long. If a class and control group are comprised of only 15 
individuals each, the impact upon absences of a single individual 
with the flu can overpower all other factors. This would be less 
likely to occur with much larger groups where influences of 
sickness would be more likely to balance out. Similarly, safety is an 
important indicator of productivity, and many workplace literacy 
programs address safety. Accidents among a group of 15 people 
during a six month period are usually rare, however, and therefore 
not likely to be of much use in determining program impact. This 
same pattern held for productivity suggestions and discipline 
measures as indicators of program impact. Neither employer 
maintained data on individual employee productivity, so those 
measures were not available. Such indicators are likely to be of 
worth when available. 

Supervisor Rctings 

Specially-constructed supervisor ratings of employee 
productivity with literacy and communication behaviors on the job 
were of greater use. Discussions with supervisors and top 
employees identified the types of literacy, problem solving, and 
communication skills considered important on particular jobs. A 
careful process of developing and revising scales to reflect these 
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discussions is available in Chapter 6 (see also Appendix G.) At the 
Cumberland plant, supervisor ratings proved to be useful in noting 
employee improvements on the job. The ratings were less 
successful at Delco. 

The reasons for this lack of success at Delco bear some 
examination. Delco supervisors generally had less education than 
the Cumberland supervisors and were less familiar with the 
concept of individual employee evaluation. Appointments between 
individual researchers and supervisors for the purpose of rating 
learners' job performance were canceled for several legitimate 
reasons. As a result, supervisors sometimes rated employees 
without someone to remind them to think carefully about each 
scale. The resulting ratings seemed to reflect a desire to complete 
the task rapidly (e.g., many workers received exactly the same 
rating on each scale). It seems advisable in the future to require 
that supervisors make ratings with a researcher asking the 
questions and encouraging careful consideration of each scale and 
each worker. 

Supervisor ratings were possible at the two pilot sites because 
learners came from highly similar jobs. Programs that draw 
learners from several different jobs may not be able to use 
supervisor ratings to assess impact on productivity. Unless jobs 
have several common tasks, it will not be possible to construct 
scales which can be used for all learners. If several different scales 
need to be constructed for several different jobs, the small number 
of learners in each job category is likely to preclude any 
meaningful statistical analysis. 

Questionnaire items related to learner literacy and 
communication practice on the job were of some use in gaining a 
picture of impact on productivity. These items are subject to some 
of the same limitations as supervisor ratings. At the two pilot sites, 
the expectation was for all workers to become involved in team 
meetings. For this reason, it was possible to have several 
questionnaire items related to such meetings. A workplace literacy 
program without such workplace commonalties would not be able 
to benefit from these questionnaire items. 

Conclusion 

This study has shown the feasibility of using a detailed impact 
assessment model with workplace literacy programs. Without 
requiring a large commitment of resources, it is possible to gather 
a great deal of information on learners’ own literacy, the literacy of 
tlieir families, and their job productivity. 





92 



T R93-2 



The results of the study indicate what can be expected of 
effective workplace literacy programs. Instruction has produced 
some improvement in all of the areas assessed, but gains appear to. 
be limited to areas directly addressed in class. There is apparently 
no transfer of learning into areas not covered by instruction. 
Because of this, it appears that program providers need to have 
clear goals for what they want to achieve in the limited time that 
learners are in class. They should also seek ways to extend this 
time beyond the classroom. One way to do this is to use on-the-job 
materials in class so that learners will be practicing outside class 
time. Also, encouraging motivation and independence is likely to 
lead learners to engage more often in literacy-related activities. 

The second phase of this study aims to determine whether the 
assessment model can be used by other workplace literacy 
programs with a minimum of assistance from project personnel. In 
addition, results from this second phase will throw more light on 
the conclusions reached here in the pilot assessment. 
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AND 

iNSTRUaiONS FOR CUSTOM DESIGNING 
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Interview 

What modifications are needed? 
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The Interview protocol that follows addresses learners' beliefs, practices, 
processes and plans related to literacy activities. Most programs can use 
the supplied questions concerning: 

• beliefs 

• practices 

• plans 

without any modifications. 

For the process section, job-specific modifications are required to 
determine how well employees read material from a particular workplace. 
This involves selection of reading materials which are key to performance at 
that workplace. These will be used to develop three job reading scenarios. 
We recommend that you select: 

• prose material (e.g., a newsletter article) 

• a graph (e.g., a key graph or chart) 

• a procedure (e.g.. an instruction sheet or job aid) 

Guidelines below provide directions for developing process, factual, 
inference and application questions for each job reading scenario. 
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Personal Information: 



INTERVIEW 



Name: Date:, 

What class are you in? 

Job you do 



I’d like to ask you some questions about reading, writing, and education. 
The answers to these questions will give us an idea of the way reading and 
writing are used here. 



Beliefs 

1. Describe someone you know who is good at reading and writing. What 
makes you choose this person? 



2. How good do you consider yom'self to be at reading and writing? What 
makes you think so? 



3. Describe how you would like to be in terms of reading and writing. [Probe 
: Could you give me some examples?) 
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Practices 
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1. Tell me the sorts of things you read and write away from work during a 
normal week. (For probe, ask: "Can you give me more examples?") 



2. TeU me the sorts of things you read and write on the job during a 
normal week. [Use probe above for more examples.) 
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Instructions for Custom-Designing 
Process : article, graph, and procedure/job aid 






(For Interview, following pages) 
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Process : Article Example 

Competitor Close Up 

1. 1 am going to show you a newspaper article about your industry. 

Explain to me how you would read this story in order to find out what 
the writer thinks. 

{Show attached story: "Competitor Close Up"). 

Describe what you would look at. What would you be thinking about? 
How would you go about reading this story? ^Vhat would you do first, 
then next, titien next? 



2. (easy factual question) 

How many employees does ASMO have in Statesville? 

(Answer: 400. Listed in article . ) 



3. (harder factual question) 

What is the only company that does not mention customers? 

(Answer: BG Automotive Motors, Inc. Requires the interviewee to look at 
all "customers" in the article . ) 



4. (easy inference question) 

From the information provided about products, what do all four 
companies have in common? 

(Answer: All of them make some sort of motor. Requires the interviewee 
to search for commonalities not readily apparent . ) 
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Process : Article Example (cont.) 



5. (harder inference question) 

Which of the companies listed is closely related to Japan and 
why do you think so? 

(Answer: ASMO or Jideco. Each has Japanese plants listed and 
each sells to many Japanese affiliates and main customers. 

Requires looking at two pieces of ii\forrnatk)n and drawing deductions 
based on what is providedL ) 



6. (harder application question) 

What company makes products closest to your job at this 
facility? Why do you say so? 

(Answer: Relate a product on the list to what the employee makes. 
Requires the employee to sort through the information and then to 
cipply it to his/her background knowledge . ) 



7. (easy application question to end the section) 

From this list, which company pays the least amount to its 
workers? How does this relate to your wages at Delco? 

(Answer: ASMO. It's more or it’s less than what I get paid here. 
Requires the employee to apply the information to his/her background 
knowledge, but allows him/her to contribute more . ) 
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Competitor 
A Year in 

Throughout the year, the Delco Doings 
has brought you profiles on the companies 
trying to take a bite out of our business and 
our profits. Sometimes there were success 
stories, when Rochester Operations met the 
challenge and came out on top. Other times 
we had to face the fact that there are 
companies In Asia, Europe, and right here 
at home that are reaching the market better, 
faster or with lower prices. 

Here’s a quick recap of the competitors 
we've covered this yea \ 

ASMO. Inc. 

Location: Battle Creek. Michigan; Statesville. 
North Carolina; Kosal City, Japan. 

Affiliate: Nippondeso 

Products: wiper ^sterns, windshield washer 
systems, power window lifts, antennas, 
retractable and blower motors. 

Main Custom<;rs: Nippondeso. Ford. 

Chrysler, General Motors, and every 
Japanese transplant except Nissan. 

Number of Employees: Battle Creek.. 130; 
Statesville. 400. 

Total Wage and Benefit Cost/Houn $9.58 
Jldeco 

Location: Bardstown, KentuclQ^;Yokohama 
City, Japan; Nine production facilities 
throughout Japan 

Affiliates: Hitachi (24%). Nissan (21%) 
Products: wipers, ^ansmissions. reservoirs 
arms and blades, wiper motors, and others. 
Control— wiper switches and others. 

Motors — power seat sliders, power window, 
door lock, blower and engine cooling 
motors and others. 

Accessories: air compressors, power window 
kits, door locks, raln-senslng Intermittent 
wiper controls and others. 

Main Customers: Nissan, Isuzu, Honda. 
Mitsubishi. Mazda, and Suzuki. 

Number of Employees: Bardstown. 60 In 1987 
Total Wage and Benefit Cost/Houn $10.27 



Close-Up: 

Review 

Power Motion 

Location: Two plants in London. 

Ontario 

Parent: Slemans Automotive of 
West Germany 

Products: air moving motors (5.250 
armatures a day) 

Main Customen GM of US & Canada 
Number of Employees: 200 at main 
facility In London. Ontario 
Total Wage and Benefit Cost/Hour: 
$11.50 (U.S. equivalent) 

BG Automotive Motors. Inc. 

Location: Hendersonville, TN. 

Parents: Bosch Corporation and General 
Electric Company 

Products: 20 different small motors 
Including; engine cooling, modular wipers, 
door lock, seat back, head rest, sun-roof, 
washer pump, head lamp, power window. 
Number of Employees: 275 
Total Wage and Benefit Cost/Hour: 
Unknown at this time. 

Every day another company steps into 
the automotive arena ready to try to take 
away our customers. Rochester Operations 
has an extensive communication network 
to keep employees informed about our 
competitors and what we're doing to stay 
ahead. Look to Delco Doings to give you 
the information you need to help keep 
Rochester Operations competitive in the 
'90s. 



From: Delco Doings. December /January, 1991, p.2. 
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Process : Graph Example 



Production Problems 

1. I am going to show you a gr;aph. Explain to me how you would read 
this graph in order to find out what it's about. 

IShow attached graph.” Production Problems"). 

Describe what you would, look at. What would you be thinking 
about? How would you go about reading this graph? What would 
you do first, then next, then next? 



2. (easy factual question) 

Wliat is the total number of culls? 
(Answer: 149. Shown at top of graph.) 



3. (harder factual question) 

What time period is covered in this chart? 

(Answer: one week or week one in May. Shown at top of graph in 
abbreviated form.) 



4. (easy inference) 

What is the biggest problem here? 
(Answer: tear outs. Longest bar on graph.) 
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Process : Graph Example (cont.) 

5. (harder inference) 

Find 3 types of problem involving measurement. 

(Possible answers: thickness, length, width, squareness. Requires 
selectionjirom list at lejt of graph.) 



6. (easy application question) 

Pick one problem and suggest at least one cause for that problem. 

(Possible answers: For example, tear outs are caused when the wood 
gets caught in the machine and is gouged; moulder bum is caused by 
wood getting caught in the machine and being burned. Uses 
interviewee's job-related knowledge . ) 



7. (more difficult application question) 

Pick a second problem and suggest both a cause and a solution 

for the problem. 

(Possible answers: tear outs, caused when the wood gets caught in the 
machine and is gouged, can be repaired with wood filler and sanding; or 
moulder knife marks can be caused by gouging of the wood in carving it 
and can be repaired if you can get at the gouge and sand it and pro^dded 
the finish hasn't already been applied. Uses interviewee's Job-related 
knowledge in more depth.) 
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Process : Procedure/Job Aid Example 

OSHACARD 

1. The government has safety reguilations and special labels in many 
worl^laces. I am going to show you a safety card that many employees 
in America must keep in their pockets while working. Tliis card shows 
how to understand safety labels. Explain to me how you would read 
this card. 

{Show attached card , "OSHA "). 

Describe what you would look at. What would you be thinking about? 
How would you go about reading this card? What would you do first, 
then next, then next? 



2 (easy factual question) 

What should you do when you see the letter “x"? 

(Answer: Ask my supervisor. Directly explained in the text ) 



3. (harder factual question) 

What do all the symbols in “k" represent? 

(Answer: airline hood or mask, gloves, a suit and boots. 
Answers are in the text, but are more difficult to find . ) 



4. (easy inference) 

What is the most common type of protection from “A" to “K"? 
(Answer: gloves. Requires the interviewee to look through several parts 
of the text and then to generalize the information ) 
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Process : Procedure/Job Aid Example (cont.) 



5. (harder inference) 

Name all the letters which refer to severe hazards. How did you 
teU this? 

(Answer: F, H, J, K. Top of the table says "4 severe hazard; 

4 probably means 4 pictures. These letters have 4 pictures. 

Requires the interviewee to make deductions between different parts of 
the card . ) 



6. (harder application) 

If a super^sor says you are about to do a job that requires 
sanding, which protective items would you choose? 

(Answer: safety glasses and a dust respirator. Optional: gloves, 
combination dust/vapor respirator and a face shield. Requires the 
interviewee to interpret the information on the card and to relate it to a 
real-life situation . ) 



' 7. (easy application question to end the section) 

Give me two examples of how you or someone you know 
could use this card. 

C-^nswer: Must give 2 examples and list protections. This is more 
open-ended and allows the interviewee to contribute based on his/her 
Job background .. ) 
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Plans 



Now I'd like to ask you about your plans. E^lain how you se e reading 
and education as part of these plans: 



A. What are your plans for the next year? 



B. What are your plans for the next 5 Years? 



C. What are your plans for the next 10 Years? 
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Questionnaire 

What modifications axe needed? 

The Questionnaire protocol that follows addresses learners' reading abilities, their literacy 
practices at work and away from work, and the literacy activities of their families. Most programs ca 
use the supplied questions concerning: 

• literacy away from work 

• literacy at work 

• family literacy 

without any modifications. 

The section on self>rating of reading ability has 15 questions, 10 of which should apply to mos 
industries and thus need no changes. However, the last 5 items should be site-specific reading 
materials, such as warning labels, route sheets, product lists, etc. Actual names may differ from sits 
to site. 

When you choose these last 5 Items, select a mix of: 

• prose and graphic materials (e.g., a note from a 
supervisor, and a blueprint) 

• ea^ and ddflcult reading materials (e.g., simple 
suggestion forms and more complex benefit 
information) 
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QUESTIONNAIRE 

Name: Age: Sex: 

Education: (furthest year in school) Training 

Marriage Status: 

Children: (number) (ages) 

Practices: Self rating reading ability 

I. 1. First check only the things you’ve read in the past month. 

2. Now go back and rate your ability to read the items you’ve checked. 

poor excellent 

_ local newspapers 1 2 3 4 5 

_ classified ads 1 2 3 4 5 

_ telephone bills 12 3 4 5 

_ TV guide listings 1 2 3 4 5 

_ magazines 1 2 3 4 5 

poor excellent 

_ training guides 1 2 3 4 .5 

_ paycheck st ubs 1 2 3 4 5 

_ company n ewsletters 1 2 3 4 5 

_ benefit information 1 2 3 4 5 

_ graphs and charts 1 2 3 4 5 

poor excellent 

_ 1 2 3 4 5 

— 1 2 3 4 5 

- 1 2 2 4 5 

- 1 2 2 4 5 

— 1 2 3 4 5 
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Instructions for Custom-Designing 
Practices : Self rating reading ability 

(For Questionnaire, preceding page) 





INSTRUCTIONS 

1. This component includes ten standard items followed 
by five examples of site specific items. 

2. In the site specific section, replace example items 
with items key to your workplace. 

3. Have a mixture of prose and document/ chart items. 

4. These should cover a range of reading difficulties. 

5. Possible sources of materials: 

• warning labels, • production quotas, 

• parts lists, • product lists 



• department inventories. 




EXAMPLE : 

poor 



blueprints i 

route sheets l 

notes from supervisor 1 
suggestion forms 1 

inventory gnaphs 



excellent 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 



1 



2 



3 



4 



5 



Practices: R eading frequency 

Please check the number of times you have done the following: 

1. In the last 7 days how many times have you used a TV guide 
listing to select programs? 

0 i 2 3 4 5 6 7 8 9 

2. In the last 7 days how many times have you read a newpaper? 

0 i 2 3 4 5 6 7 8 9 10+ 

3. In the last 7 days how many times have you read a magazine? 

0 i 2 3 4 5 6 7 8 9 Io+ 

4. In the last 7 days how many times have you read a book for pleasure? 

0 i 2 3“^ 4 5 6 7 8 9 10+ 

5. In the last 7 days how many times have you read the following types of 
books? 



mystery: 


_times 


how-to books: 


times 


novels: 


times 


factual books: 


times 


poetry: 


times 


encyclopedia: 


times 


Bible: 


times 


comic books: 


times 


other types: 




times 






times 
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Prabtices: R eading frequency (cont.) 



6. How often do you make a shopping list before you go to the store? 
never occasionally often always 



7. When you’re waiting in an office, how often do you read magazines? 

never occasionally often always 

8. Do you subscribe to any magazines? yes no 

If yes, which ones? 



9. How many different magazine titles do you have in your home? 



0123456789 10+ 



10. How many books are in your home, either owned or borrowed? 



1 

I 

I 

I 

I 

I 

I 



0 10 20 30 40 50 60 70 80 90 100+ 
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Practices: Literacy at work 



Please circle the number w hich best describes you in the situations 
below: 

(1) You just listen in team or department meeting discussions. 

very like me 1 2 3 4 5 very unlike me 

(2) You talk a lot in team or department meetings, asking questions or 
sharing ideas. 

very like me 1 2 3 4 5 very unlike me 

(3) Your ideas are often discussed in team or department meetings. 

very like me 1 2 3 4 5 very unlike me 

(4) You wait for others to talk about written information, just to be sure 
what is in it. 

very like me 1 2 3 4 5 very unlike me 

(5) You look for printed directions to help figure out what to do when a 
problem arises. 

very like me 1 2 3 4 5 very unlike me 

(6) You often have trouble reading paperwork from management. 

very like me 1 2 3 4 5 very unlike me 

(7) When the booklet about new health benefits arrived, you r^^ad it 
carefully. 

very like me 1 2 3 4 5 very unlike me 



Practices ; Family Literacy 



Only answer the following questions if you have a child between the ases of 3-17 at 
honiCt 

Please answer for your youngest child in this age group and please fill in 
only one answer per question: 

1. This child is years old. 

2. In the last 7 days how many times has your child looked at or read 
books or magazines? 

0 1 2 3 4 5 6 7 8 9 T6+ 

3. In the last 7 days how many times has your child seen you reading or 

writing? ^ 

0 i ¥ 3 4 5 6 7 8 9 lo^ 

4. In the last 7 days how many times haye you helped your child with 
homework and/or with school projects? 

0 T~ 2 3 4 5 6 7 8 9 loT 

5. In the last 7 days how many times have you read/looked at books with 
your child or listened to him/her read? 

0 T 2 3 "4 T 6 7 8 9 

6. In the last 7 days how many times has your cMd asked to be read to? 

0 1 2 3 4 5 6 7 8 9 loT 

7. In the last 7 days how many times has your child printed, made letters or 
written? 

0 i 2“^ 3 4 5 6 7 8 9 
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Practices ; Family Literacy (cont.) 



8. In the last month how many times has your child gone to a public 
library? 



0123456789 10+ 



9. In the last month how many times have you participated /helped out 
in your child's school? 



0123456789 10+ 



10. In the last month how many times have you hxmg up or displayed 
your child's reading and writing efforts? 



0123456789 10+ 



1 1. In the last month how many times have you bought or borrowed 
books for your child? 



0123456789 10+ 



12. (Please check only one.) 

I expect my child to finish at least: 

^6th grade 9th grade ^high school 

_two-year college 4-year college or more 



o 








TR93-2 



Appendix C 



Cloze Test Samples 

AND 

iNSTRUaiONS FOR CUSTOM DESIGNING 



-i 0 

NATIONAL CENTER ON ADULT LITERACY 



I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 



Cloze Exercise 

The cloze procedure is based on the psychological principle of closure, 
which is the human tendency to recognize and complete a pattern or 
sequence. It involves replacing missing words in a reading passage. This 
procedure can assess the ability of employees to comprehend the passage. 
Cloze test scores correlate very highly with standardized reading test scores. 
Cloze tests can be made from local workplace materials. 



CLOZE PROCEDURE 

1 . Select a job relevant passage of 1 50-200 words. 

2. Leave the first and last sentences intact. 

3. Starting with the second sentence, omit every fifth 
word. This will give about 25 blanks. Replace ail 
omitted words with the same sized blank line, 
13-15 spaces is typical. 

4. Employees are to read the passage and to fill the 
blanks with their best guess at the word removed. 

5. Instructions for the cloze test should suggest 
employees read the entire text before attempting 
to fill in the blanks and should encourage 
employees to answer all questions even If they 

have to guess. It Is rare for anyone to know more 
than 50% of the blanks. 

6. Avoid controversial or emotional topics, and 
topics requiring technical knowledge. Scores for 

such materials are less valid. 
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Date 

CLOZE Exercise 



In a cloze exercise, you try to guess which words are missing. For 
example, in the sentence below, a word is missing. 



She looked before she 



the street. 



A good guess for the missing word is "crossed." 

She looked before she crossed the street. 

In the story below, try to guess and replace the missing words. Don't 
expect to get them all. Some are nearly impossible. 

G.M Designs Safety for All Ages 

We all like to think about the old days. Life seemed simpler and, in 

some ways, better then. But when it comes to , the good 

old days offer the same degree safety as 

today's cars trucks. Advancements in technology 

the G.M. vehicle you today among the safest 

the world. Each G.M. and truck is backed 

thousands of dedicated men women who 

care about safety of their customers. , as 

G.M. customers themselves, have a stake in 

G.M. vehicles the highest quality and reliability. 

And you're wondering if safety improved 



in recent years. 



, this: The classic 1955 



would 



require more than major changes or additions 

hundreds of incremental changes be as safe 



as 



vehicles. 



From: Kilbom, C. GM Today tNovember/December. 1990). page 1. 
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Cloze Exercise Answer Key 
G.M Designs Safety for All Ages 

We all like to think about the old days. Life seemed simpler 
and, in some ways, better then. But when it comes to automoSiCes . the 
good old days didn't o ffer the same degree of safety as 
today's cars and trucks. Advancements in technology make 
the G.M. vehicle you 'pUTcftOse today among the safest in the world. 
Each G.M. COT and truck is backed t housands of 

dedicated men and women who care about the safety of 
their customers. ^bld as G.M. customers themselves, they 

have a stake in mokiny G.M. vehicles the highest in quality 
and reliability. 

And if you're wondering if safety has 
improved in recent years, consider this: The classic 1955 
Chevrolet would require more than 60 major changes or 
additions and hundreds of incremental changes to be as 
safe as today's vehicles. 

m 
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CLOZE Exercise 

In a cloze exercise, you tiy to guess which words are missing. For 
example, in the sentence below, a word is missing. 

She looked before she the street. 

A good guess for the missing word is "crossed.” 

She looked before she crossecC the street. 

In the story below, try to guess and replace the missing words. Don't 
expect to get them all. Some are nearly impossible. 

Two more teams on the self-directed lournev 
Our workplace is taking on more change daily. So are the skills that 
all our employees must have in order to change with it. It is getting to be 

essential each day that skill gaps be filled 

our small business can a source of competitive 



The changes all companies expect over the next - 

a shrinking labor force, demand for workers in 

jobs, and increasingly competitive markets -- will require 

businesses all sizes to strengthen employee skills 

and training 

We believe that our firm can remain competitive the 

large firm by a more flexible training education 

program. We hope be better than the firm in adapting 

an previous training experiences to company’s needs. 

Two more are now involved in self-directed team 

training. They are Green Team and the Team. They 

Join the Orange Team, which completed their sessions last year. 
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Cloze Exercise Answer Key 

Twamore teams on the self-directed journey 

Our workplace is taking on more change daily. So are the skills that 
all our employees must have in order to change with it. It is getting to be 

more essential each day that the skill gaps be filled SO 

our small business can remain a source of competitive Strength . 

The changes all companies can expect over the next decade 
“ a shrinking labor force. more demand for workers in technical 
jobs, and increasingly competitive world markets — will require 
businesses of all sizes to strengthen their employee skills 
and training Programs . 

We believe that our Small firm can remain competitive with the 
large firm by having a more flexible training and education 
program. We hope tO be better than the laroe firm in adapting 
an employee’s previous training experiences t o the company’s 
needs. 

Two more teams are now involved in self-directed work team 

training. They are the Green Team and the White Te am They 

join the Orange Team, which completed their sessions last year. 
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CLOZE Exercise 

In a cloze exercise, you try to guess which words are missing. For 

example, in the sentence below, a word is missing. 

She looked before she the street. 

A good guess for the missing word is "crossed." 

She looked before she crossed the street. 

In the writing below, tiy to guess and replace the missing words. Don't 

expect to get them all. Some are nearly impossible. 

Cumberland Safety Rules 

1. For your welfare, all Injuries, no matter how slight, incurred on Company premises 
must be reported Immediately to your supervisor. The services of a physician 

available and will be as required. Failure to 

such Injuries may cause difficulties and could affect 

Workingmen’s Compensation benefits. 

2. Wear and shoes suitable to work. Open toed or 

top shoes are not _____ . Shorts are not permitted. 

are required. 

3. Dust your only with an air equipped with an air 

nozzle. 

4. Keep fire equipment Its proper place and all fire rules. 

5. Learn lift properly. Keep your straight and use your 

to avoid strain. 

6. All guards should be kept place. Unsafe machine 

guards be reported to your at once. No guard 

be removed without the of your supervisor. 

7. Do repair machinery when It In operation. Stop It and 

fix the switch so that It cannot be accidently turned on. 
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Cloze Exercise Answer Key 
Cumberland Safety Rules 

1. For your welfare, all Injuries, no matter how slight. Incurred on Company premises 
must be reported immediate^ to your supervisor. The services of a physician are 

available and will be obtained as required. Failure to report such 

Injuries may cause medical difficulties and could affect vour Workingmen’s 
Compensation benefits. 

2. Wear clothing and shoes suitable to vour work. Open toed or canvas 
top shoes are no t permitted . Shorts are not permitted. Shirts are required. 

3. Dust your clothes only with an air hose equipped with an ai r restricting 

nozzle. 

4. Keep fire equipment is proper place and obey all fire rules. 

5. Learn to lift properly. Keep vou r back straight and use vou r leas to 
avoid strain. 

6. All machine guards should be kept m place. Unsafe machine guards should 

be reported to your supervisor at once. No guard should hp removed without 
the permission of your supervisor. 

7. Do not repair machinery when it is I n operation. Stop It and fix the switch so 
that It cannot be accidently turned on. 
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Family Literacy Focus Group Interview 



This interview form is designed to be used with a group of 
learners as the basis for a discussion about family literacy. It has 
been found that the comments of one member of the group will 
stimulate the thoughts of others, producing a wider range of ideas 
than will individual interviews. 



i;<‘i 

NATIONAL CENTER ON ADULT LITERACY 




Faxsdly Literacy Focus Group Interview 



1. Why do you think some children learn to read and write well in school 
and others don't? 



2. Do you think there is an)dhing parents might do to help their children 
learn to read and write better? 



3. Do you keep any reading or writing materials at home? i.e., letter 
blocks, flashcards, paper, pens, chalkboard, books, magazines, 
comics, cassettes with books, encyclopedia, dictionary, newspapers, 
etc) 
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4. Do you do any reading or writing activities with your children? (i.e., visit 
library, hear stories, read to them, watch educational television, look at 
magazines or books with children, point out words to them, play school, 
show them how to read or write, etc) 



5. At home, do your children see you doing any reading or writing? (i.e. 
books, magazines, papers, recipes, directions, letters, lists, notes, etc.) 



6. What activities are you involved in at your child's school? 

(i.e., parent/teacher meetings, school fund-raisers, committees, notes or 
letters, informal talks when collecting child, assist in classroom, help 
child read at home, etc) 
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7. Have you begun anytJiing new related to reading and writing since you 
started classes here? 

a. Materials 



b. Activities 



c. Modelling 



d. School 
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Classroom 
Observation Form 
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Classroom Observation Form 

I 

S The classroom observation form serves as a guide for recording notes 

about the activities actually occurring in the classroom. It is divided into 

■ columns reflecting the time in five minute intervals, the actual activities of 

both teacher and student, and comments about the overall class. The form 
suggests items the observer might wish to note. 
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Time 

0 



Student Activity 



Teacher Activity Comments 



05 

10 

15 

20 

25 

30 

35 

40 

45 

50 

55 



Make note of time spent by students actually reading or doing things. Also note time 
learners spend listening to the instructor. When learners are in small groups or 
working individually should be mentioned. Special note should be made when the 
instructor or a student demonstrates how to do something. 
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ESL Checkust 
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The E^SL checklist is designed for teachers to reflect upon the level of 
competence each student is demonstrating. Teachers will be able to note 
individual areas of strength and weakness. This form is helpful both in 
planning instruction and in suggesting areas for the student to practice on 
outside of the workplace. 
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ESL Benchmarks and Ratings* 

Learner Name Teacher Name 

Date of Rating 

For each item, rate the learner: 

3 s can do this as well or nearly as well as a native speaker 
2 s can usually manage to do this, but sometimes has trouble 
1 s can only sometimes manage to do this adequately 
O s cannot do this 

Beginner Level 

Briefly describes feelings about work 

Briefly describes feelings about other life areas 

Follows simple directions 

Asks for clarification if something is not imderstood 

Reads alphabet hi English 

Word recognition: 

has access to dictionaiy/imderstands dictionary use 

uses dictionary 

uses roots, prefix, suffix 

uses context 

Looks up simple information (phone book, dictionaiy) 

Reads simple signs 

Begins short journal entries 



* Modified from Bronstein, E. (1991) Benchmarks and student learning 
profile for the workplace E:SL pro^am of the Labor Education Center 
at Southeastern Mass. University 



Intermediate Level 
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Oral 

Discusses feelings about work with some elaboration 

Discusses feelings about other life areas with some elaboration 

Gives/follows instructions at work 

Gives/follows instructions in other life areas 

Asks for clarification if something is not understood 

Discusses industrial specific diseases/illnesses 

Describes/reports dangerous conditions 

Offers suggestions to supervisor 

Reading 

Uses dictionary ( bilingual English-English) 

Locates own reading material in newspapers 

Understands literal level of text 

Infers information not explicitly stated 

Draws conclusions from reading 

Writing 

Fills out more complex forms 

job application social security insurance 

other application forms (library card, courtesy card, credit 

card) 

Writes short notes/memos ( at work 

out of work) 

Writes journal entries (dialogue journal) 

Uses correct punctuation 
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Advanced Level 

Oral 

Discusses feelings with more elaboration 

Asks for clarification if something is not understood 

Gives /follows more complex directions 

Understands and can discuss basic worker rights 

Understands and can discuss key contract sections 

Reading 

Uses index/table of contents ^ 

Locates own reading material in newspaper 

Locates own reading material in encjrclopedia or other reference 

Reads short pieces in newspaper and simple magazines 

f 

Reads short pieces in flyers, notices, factsheets 

Reads short self-selected material at home 

Reads longer pieces in books or longer articles 

Writing 

Fills out more complex forms 

job application social security insurance 

other application forms (library card, courtesy card, credit 

card) 

Writes short notes/memos ( at work out of work) 

Observes differences in tone/register between formal and informal 

writing 

Writes longer journal entries /responds to entries 
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Supervisor Rating Scales 
Examples and iNSTRuaioNs 
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Developing Supervisor Ratings 

It is best to develop ratings of employee job performance 
together with supervisors and possibly key employees. 

1. First ask supervisors to describe bow top performers use 
information on the job. 

Encourage them to think of specific workers who are top 
performers. A supervisor might say, for example, that a top 
performer reads charts and responds with his own analysis, 
or sets machines correctly and checks settings thoroughly, 
or completes all job-related paperwork and tries to improve 
procedures. Continue to probe until you feel reasonably 
satisfied you have a complete list. FVom this list, you can 
identify important areas (i.e. communication, problem 
solving, paperwork, etc.) 



Next ask supervisors to go through a three-step process in 
fieshing out these areas. The order of these steps is important. 

2. Ask supervisors to: 

a. describe the behavior of the top performers first; 

b. then, describe the behavior of the bottom performers; 

c. last, describe the average performer. 

Tliese behaviors will be used to provide descriptions and 
anchors for ratings. In relation to paperwork, for example, 
supervisors might agree on the following descriptions: 

Top: completes all job-related paperwork and tries to 

improve procedures; 

Bottom: intimidated by job-related paperwork and does it 
poorly; 

Average: does job-related paperwork but simply keeps pace. 
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As supervisors develop these descriptions, new areas and 
categories may emerge. The supervisors may give examples 
related to problem-solving or to machine setting, or some 
other area. These may later become additional rating 
scales. 



3. Once the descriptions of top, bottom, and average performances 
are 

completed, work with supervisors to develop acceptable labels 
for the categories. 

For example, labels might include items like machine setting, 
paperwork, communication, responsibility, and 
problem-solving. 



4. After this discussion, you will draft a rating scale and submit it to 
the supervisors for comment and possible revision. Sometimes 
during revision, complex scales split to become two separate 
scales. 

Examples of scales appear on the following pages. 
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Employee Assessment - Overall Rating 

Please rate each employee on a scale of 1 - 10 for each aspect 
below. 


• An average employee would be rated 5. 

• A top employee would be rated 8 or higher. 

• A bottom employee would be rated 2 or lower. 




EMPLOYEE 




DATE 




RATER 


COMMUNICATION 






Bottom 


Average 


Top 


won't speak; 
can't express self; 
nervous; won't 
shake hands 


open, relaxed 
communicator; 
good listener 
and responder 


processes 
information 
and responds 
with own analysis 


12 3 4 


5 6 7 


8 9 1C 


CONCERNS. PROBLEM-SOLVING 




Bottom 


Average 


Top 


doesn't consider 
alternative solutions; 
makes irrelevant 
suggestions; 
never thinks of 
consequences 


can suggest 
solutions, but 
not work through 
them in detail 


suggests solutions 
and analyses 
consequences, 
including a 
timeline 
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HANDLING CONFLICT 



Bottom 


Average 


T^ 


antagonistic: 
turns back on others: 
makes abrupt denials 
and impolite comments 


cooperates with 
others most of 
the time, but some 
antagonism 


empathetic: 

cooperative: 

consistent 

attitude 


12 3 4 


5 6 7 


8 9 10 


SELF-ESTEEM 






Bottom 


Average 


Top 


shy: uncertain: 
overwhelmed by 
life’s problems 


some confidence 
in self: but life 
not really 
under control 


confident: 
usually in control 
of life and of 
most situations 


12 3 4 


5 6 7 


8 9 10 


SETTING GOALS 






Bottom 


Average 


Top 


unable to plan 
ahead and 
set goals 


some short-term 
planning and 
goal setting 


clear plans for 
future: definite, 
reachable goals 
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COMMITMENT 



Bottom 


Average 


Top 


lacks motivation; 


some commitment; 


conscientious; 


no interest in 


but just doing a 


committed to 


company goals 


competent job 


company goals 


12 3 


4 5 6 7 


8 9 10 



RESPONSIBILITY 






Bottom 


Average 


T0£ 


has to be told what 
to do and checked on 


can be left to carry 
out routine work 


dependable; takes 
responsibility 
for own work 


12 3 4 


5 6 7 


8 9 10 



INITIATIVE 






Bottom 


Average 


Top 


ignores machine 
errors and lets 
them build up 


realizes machine 
errors and 
attempts 
immediate 
solution only 


monitors machine 
errors and deals 
with them 
through the team 


12 3 


4 5 6 7 


8 9 10 
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PAPERWORK 



Bottom 


Average 


Top 


intimidated 
by job-related 
paperwork and 
does it poorly 


does Job-related 
paperwork, 
simply 
keeping pace 


completes all 
job-related 
paperwork and 
tries to improve 
procedures 


12 3 4 


5 6 7 


8 9 10 


MACHINE SETTING 






Bottom 


Average 


T0£ 


unable to set 
machines correctly 


usually sets 
machines correctly, 
but doesn't always 
clieck settings 


sets machines 
correctly and 
checks settings 
thoroughly 
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Tabular Data 
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Glossary of Variables 



In the tables that follow, the variables are given brief descriptions which 
may not always be entirely clear. So this ^ossary provides a fuller 
explanation for those variables that require it. 

BELIEFS AND PLANS 



Literacy self-rating 
Change in literaqr 
self-image 


Learner self-rating of literacy level (on scale 1-5) 
Holistic judgement of learner's change in literacy 
self-image (on scale -1, 0, +1) 


Change in plans for 
1, 5, 10 years 
Change in plans for 
education 


Holistic judgement of learner’s change in plans for 
1, 5, 10 years (on scale -1, 0, +1) 

Holistic judgement of learner's change in plans for 
reading and education (on scale -1, 0, +1) 


PRACTICES 

Reading/writing 
away from work 
Reading/writing 
at work 
Items read (in 
20 item list) 


Count of types of reading/writing away from work 
in last week 

Count of types of reading/writing at work 
in last week 

Count of items from given list read in last month 


Frequency of 
reading activities 
Ownership of 
subscription 


Sum of 6 frequency ratings of literacy activities 
(each on scale 1 = never to 5 = eveiyday) 

Sum of book ownership and magazine 


reading materials 


( on scale 1 = 1-5 to 8 = 50+) 


Self-rating on 
meetings 


Learner self-rating on their participation in 


talking in meetings 
Self-rating on 
ide^ discussed 
Self-rating on 
asking for help 


(on scale 1-5) 

Learner self-rating on how much their ideas are 
discussed in meetings (on scale 1-5) 

Learner self-rating on how much they ask for help 
at work (on scale 1-5) 


PROCESSES 

Total process 
responses 


Count of all responses to process question 


Focus 

responses 

Strategy 

responses 

Topic 

responses 


Count of responses to process question involving 
points of focus (e.g. title, bold print) 

Count of responses to process question involving 
reading strategies (e.g. skim, read through) 
Count of responses to process question involving 
topics of interest (e.g. products, wages) 



* In each pair of values, mean is above and standard deviation is below. 
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Delco: Technical Prepaxation Class (n = 14) 



BELIEFS & PLANS 


Pre-test Post-test 

mean/s.d.* mean/s.d. 


Change 

mean/s.d. 


Significance 


Literacy 


3.357 3.929 


.571 


p<.05 


self-rating 


1.082 .829 


.938 




Change In literacy 




0.5 


p<.01 


self-image 


(Holistic 


.65 




Change in plans 
for 1 year 


Judgements 

ojchaange: 


0.429 

.646 


p<.05 


Change in plans 




0.357 


p<.05 


for 5 years 


nopre- 


.633 




Change in plans 
for 10 years 


and post- 
scores) 


0.0 

.679 


n.s. 


Change in plans 
for education 




- .143 
.949 


n.s. 



PRACTICES 



Reading/ writing 


4.786 


6.571 


1.786 


p<.05 


away from work 


2.082 


2.709 


2.86 


Reading/ writing 


2.846 


2.615 


- .231 


n.s. 


at work 


1.725 


2.293 


3.516 




Items read (in 


17.308 


18.846 


1.538 


n.s. 


20 item list) 


4.644 


2.734 


3.799 




Frequency of 


16.308 


16.923 


.615 


n.s. 


reading activities 


3.093 


3.201 


1.502 




Ownership of 


5.154 


5.154 


0.0 


n.s. 


reading materials 


2.035 


1.951 


1.354 




Self-rating on 


2.769 


3.231 


.462 


p<.05 


talking in meetings 


1.301 


1.235 


.776 


Self-rating on 


2.385 


3.231 


.846 


p<.05 


ideas discussed 


1.325 


1.092 


1 463 


Self-rating on 


1.615 


1.615 


0.0 


, n.s. 


asking for help 


.768 


.768 


1.08 


tj 



In each pair of values, mean is above and standard deviation is below. 
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Delco: Technical Preparation Class (cont.) 



PROCESSES 


Pre-test 

mean/s.d.* 


Post-test 

mean/s.d. 


Change 

mean/s.d. 


Significance 


Total process 


4.214 


5.286 


1.071 


n.s. 


responses 


1.424 


1.858 


2.526 




Focus 


1.786 


1.786 


0.0 


n.s. 


responses 


1.188 


1.251 


.961 




Strategy 


1.143 


1.357 


.214 


n.s. 


responses 


.864 


.842 


1.051 




Topic 


1.286 


2.143 


.857 


n,s. 


responses 


1.437 


1.748 


2.143 




Article question 


.929 


1.0 


.071 


n.s. 


(easy factual) 


.267 


0.0 


.267 


Article question 


1.5 


1.714 


.214 


p<.05 


(harder factual) 


.519 


.469 


.426 


Graph question 


4.857 


4.857 


0.0 


n.s. 


(easy factual) 


.363 


.363 


.392 


Graph question 


4.571 


4.643 


.071 


n.s. 


(easy factual) 


.756 


.633 


.917 


Graph question 


4.071 


3.714 


- .357 


n.s. 


(haitier factual) 


1.207 


1.139 


1.008 


Graph question 


2.071 


2.786 


.714 


p<.01 


(inference) 


1.072 


.893 


.994 


Job aid question 


.929 


1.0 


.071 


n.s. 


(easy factual) 


.267 


0.0 


.267 


Job aid question 


2.429 


2.643 


.214 


n.s. 


(harder factual) 


.646 


.497 


.893 


Job aid question 


2.643 


2.857 


.214 


n.s. 


(inference) 


.745 


.535 


.975 


Cloze test score 


10.857 

2.685 


12.429 

3.131 


1.571 

2.377 


p<.05 



• In each pair of values, mean Is above and standard deviation Is below. 
NATIONAL CENTER ON ADULT LITERACY 
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5b* 



/ 



Delco: Tecbnlcal Preparation Control (n = 12] 



BBUBPS & PLANS 


Pre-test 


Post-test 


Change 


Significance 




mean/s.d.* 


mean/s.d. 


mean/s.d. 


literacy 


3.583 


3.5 


- .083 


n.s. 


self-rating 


.669 


.674 


.515 




Change in literacy 






.083 


n.s. 


self-image 


(Holistic 




.515 




Change in plans 


Judgements 




- .167 


n.s. 


for 1 year 






.718 




Change hi plans 


of change: 




- .333 


n.s. 


for 5 years 


no pre- 




.492 




Change in plans 


and post- 




- .083 


n.s. 


for 10 years 


scores) 




.515 




Change in plans 






2.25 


n.s. 


for education 






.866 




PRACnCBS 


Reading/writing 


4.75 


4.583 


- .167 


n.s. 


away from work 


1.42 


1.564 


2.29 




Reading/writing 


3.0 


3.083 


.083 


n.s. 


at work 


.853 


1.832 


1.505 




Items read (in 


18.75 


18.583 


- .167 


n.s. 


20 item list) 


2.34 


2.644 


3.81 




Frequency of 


18.167 


16.583 


- 1.583 


n.s. 


reading activities 


2.082 


1.676 


2.644 




Ownership of 


4.917 


4.833 


- .083 


n.s. 


reading materials 


2.019 


2.038 


1.165 




Self-rating on 


3.333 


3.083 


- .25 


n.s. 


talking in meetings 


1.614 


1.311 


2.094 




Self-rating on 


3.5 


3.25 


- .25 


n.s. 


ideas discussed 


1.314 


1.055 


1.138 




Self-rating on 


1.667 


2.25 


.583 


n.s. 


asking for help 


1.231 


1.485 


1.379 





* In each pair of values, mean is above and standard deviation is below. 
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Delco: Technical Preparation Control (conU 



I 



I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 



PROCESSES Pre-test 

mean/s.d.* 



Total process 


5.75 


responses 


1.765 


Focus 


1.917 


responses 


.9 


Strategy 


1.583 


responses 


1.443 


Topic 


2.25 


responses 


1.712 


Article question 


1.0 


(easy factual) 


0.0 


Article question 


1.417 


(harder factual) 


.515 


Graph question 


4.583 


(easy factual) 


.515 


Graph question 


3.917 


(easy factual) 


1.311 


Graph question 


2.75 


(harder factual) 


1.865 


Graph question 


2.75 


(Inference) 


1.138 


Job aid question 


1.0 


(easy factual) 


0.0 


Job aid question 


2.917 


(harder factual) 


.289 


Job aid question 


2.583 


(inference) 


.793 


Cloze test score 


9.583 

2.843 



Post-test 

mean/s.d. 


Change 

mean/s.d. 


Significance 


5.25 


- .5 


n.s. 


1.545 


2.393 




2.083 


.167 


n.s. 


.9 


.937 




1.0 


-.583 


n.s. 


.603 


1.676 




2.167 


-.083 


n.s. 


1.801 


1.676 




1.0 


0.0 


n.s. 


0.0 


0.0 




1.583 


.167 


n.s. 


.515 


.718 




4.333 


-.25 


n.s. 


.651 


.754 




3.917 


0.0 


n.s. . ^ 


.793 


1.044 




3.833 


1.083 


p<.05 


.835 


1.782 




2.583 


-.167 


n.s. 


.669 


1.267 




1.0 


0.0 


n.s. 


0.0 


0.0 




2.833 


-.083 


n.s. 


.577 


.289 




2.667 


.083 


n.s. 


.778 


.289 




10.417 


.833 


n.s. 


2.968 


1.801 







* In each pair of values, mean is above and standard deviation is below. 
NATIONAL CENTER ON ADULT LITERACY 
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Delco: GED Class (n=15) 



BEUEFS & PLANS 


Pre-test 


Post-test 


Change 


Significance 




mean/s.d.* 


mean/s.d. 


mean/s.d. 




Literacy 


2.9 


2.8 


- .1 


n.s. 


self-rating 


.568 


i632 


.568 




Change in literacy 






0.133 


n.s. 


self-image 


(HoUstlc 




.64 




Change In plans 


Judgements 




0.0 


n.s. 


for 1 year 






.845 




of change: 




0.2 




Change In plans 






n.s. 


for 5 years 


no pre- 




.561 




Change in plans 


and post- 




0.067 


n.s. 


for 10 years 






.704 




scores) 








Change in plEuns 






0.067 


n.s. 


for education 






.704 




PRACTICES 


Reading/writing 


3.867 


4.2 


.333 


n.s. 


away from work 


1.407 


1.424 


1.447 




Reading/writing 


2.133 


2.267 


.133 


n.s. 


at work 


1.598 


1.71 


1.642 




Items read (in . 


18.867 


19.6 


.733 


p<.05 


20 item list) 


1.807 


1.056 


1.1 




Frequency of 


17.0 


17.357 


.357 


n.s. 


reading activities 


2.746 


2.56 


2.62 




Ownership of 


5.463 


5.692 


.231 


n.s. 


reading materials 


2.634 


2.136 


1.691 




Self-rating on 


3.0 


3.267 


.267 


n.s. 


talking in meetings 


1.464 


1.58 


1.58 




Self-rating on 


3.067 


3.4 


.333 


n.s. 


ideas discussed 


1.28 


1.242 


.9 




Self-raUng on 


1.533 


1.667 


.133 


n.s. 


asking for help 


1.125 


1.113 


1.598 





• In each pair of values, mean Is above and standard deviation Is below. 
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Delco: GED Class (cont.) 



I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 

I 




PROCESSES 


Pre-test 

mean/s.d.* 


Post-test 

mean/s.d. 


Change 

mean/s.d. 


Significance 


Total process 


3.333 


5.133 


1.8 


p<.001 


responses 


1.175 


2.1 


1.821 


Focus 


1.533 


1.6 


.067 


n.s. 


responses 


.915 


.737 


.594 




strategy 


1.667 


1.133 


- .533 


p<.05 


responses 


1.047 


.743 


1.125 


Topic 


.133 


2.4 


2.267 


p<.001 


responses 


.352 


2.261 


2.187 


Article question 


.933 


.867 


- .067 


n.s. 


(easy factual) 


.258 


.352 


.258 




Article question 


1.467 


1.267 


- .2 


n.s. 


(harder factual) 


.516 


.594 


.676 




Graph question 


3.8 


3.933 


.133 


n.s. 


(easy factual) 


1.082 


.594 


1.187 




Graph question 


3.4 


3.467 


.067 


n.s. 


(easy factual) 


1.183 


.64 


1.223 




Graph question 


3.133 


2.667 


- .467 


n.s. 


(harder factual) 


1.407 


1.496 


1.356 




Graph question 


1.6 


2.067 


.467 


n.s. 


(inference) 


1.454 


1.1 


1.598 




Job aid question 


.533 


.467 


- .067 


n.s. 


(easy factual) 


.516 


.516 


.594 




Job aid question 


2.4 


2.867 


.467 


p<.05 


(harder factual) 


.828 


.352 


.915 


Job aid question 


2.667 


2.4 


- .267 


n.s. 


(Inference) 


.724 


1.121 


1.438 




Cloze test score 


7.467 


7.933 


.467 


n.s. 




1.642 


2.187 


2.232 





* In each pair of values, mean is above and standard deviation is below. 
NATIONAL CENTER ON ADULT LITERACY 
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H*vfi 



Delco: ESL Class (n = 15) 



BEUEPS & PLANS 


Pre-test 


Post-test 


Change 


Significance 




mean/s.d.* 


mean/s.d. 


mean/s.d. 


Literacy 


3.091 


3.455 


.364 


n*s. 


self-rating 


.302 


.522 


.505 




Change In literacy 






0.067 


n«s* 


self-image 


(Holistic 




.799 




Change In plans 


Judgements 




0.2 


n*s« 


for 1 year 


of change: 




.775 




Change In plans 






0.2 


n*s. 


for 5 years 


no pre- 




.676 




Change In plans 


and post- 




0.067 


n.s. 


for 10 years 


scores) 




.704 




Change In plans 






0.533 


p<.005 


for education 






.64 



PRACTICES 



Reading/ writing 


4.8 


4.6 


- .2 


n.s. 


away from work 


1.897 


1.805 


2.077 




Reading/writing 


1.867 


2.6 


.733 


n*s. 


at work 


1.506 


1.682 


1.624 




Items read (In 


16.286 


19.143 


2.857 


p<*005 


20 Item list) 


3.539 


1.657 


3.009 


Frequency of 


15.6 


16.333 


.733 


n.s. 


reading activities 


4.205 


4.624 


1.907 




Ownership of 


4.462 


4.923 


.462 


n.s« 


reading materials 


2.295 


2.326 


1.506 




Self-rating on 


2.333 


2.6 


.267 


n.s. 


talking in meetings 


1.175 


1.242 


.799 




Self-rating on 


2.5 


2.571 


.071 




Ideas discussed 


1.401 


1.016 


1.141 




Self-rating on 


1.267 


1.933 


.667 


p<.05 


asking for help 


.594 


1.033 


1.234 



* In each pair of values, mean Is above and standard deviation Is below. 
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Delco: ESL Class (cont) 



I 

I 



PROCESSES 


' Pre-test 


Post-test 


Change 


Significance 


■ 




^ "mean/s.d.* 


mean/s.d. 


mean/s.d. 


1 


Total process 


3.533 


5.0 


1.467 


p<.0005 


responses 


.99 


1.414 


1.356 


1 


Focus 


1.733 


1.667 


- .067 


n.s. 




responses 


.884 


.724 


.961 




1 


strategy 


1.533 


1.667 


.133 


n.s. 


■ 


responses 


1.125 


1.047 


1.685 




1 


Topic 


.267 


1.667 


1.4 


p<.005 


1 


responses 


.594 


1.543 


1.765 


1 


Article question 


.867 


.867 


0.0 


n.s. 




(easy factual) 


.352 


.352 


.378 


1 


Article question 


.8 


1.333 


.533 


p<.05 


■ 


(harder factual) 


.676 


.617 


.99 


1 


Graph question 


3.733 


3.733 


0.0 


n.s. 


(easy factual) 


1.1 


1.1 


1.363 


1 


Graph question 


3.0 


3.467 


.467 


n.s. 


■ 


' (easy factual) 


1.195 


1.125 


1.642 


1 


1 Graph question 


2.4 


3.4 


1.0 


p<.05 


1 


1 (harder factual) 


1.844 


1.183 


1.813 


■ 


1 Graph question 


1.733 


2.333 


.6 


n.s. 


1 


1 (Inference) 


1.335 


1.345 


1.454 


1 


Job aid question 


.667 


.733 


.067 


n.s. 


■ 


1 (easy factual) 


.488 


.458 


.458 


1 


I Job aid question 


2.2 


2.533 


.333 


n.s. 


1 


1 (harder factual) 


1.014 


.99 


.9 


1 


1 Job aid question 


1.6 


2.533 


.933 


p<.005 


1 


1 (Inference) 


1.352 


1.06 


1.163 


1 


Cloze test score 


6.467 


7.0 


.533 


n.s. 




1 


2.615 


3.229 


2.642 





I 



• In each pair of values, mean is above and standard deviation Is belov', 
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Cumberland: Communications and Collaboration Class (n = 21) 



PRACTICES Pre-test Post-test Change Significance 

mean/s.d.* mean/s.d. mean/s.d. 



Self-raUng on 


3.2 


2.933 


- .267 


n.s. 


talking In meetings 


1.424 


1.335 


1.033 




Self-rating on 


3.0 


3.333 


.333 


n.s. 


ideas discussed 


1.464 


1.345 


1.397 




Self-rating on 


1.214 


1.643 


.429 


p<.05 


asking for help 


.579 


1.008 


.646 



SUPERVISOR RATINGS 



Commimication 


2.762 


5.048 


2.286 


p<.0001 




.944 


1.024 


.644 


Concerns, 


2.667 


4.857 


2.19 


p<.0001 


problem-solving 


.796 


1.014 


.928 


Handling 


3.0 


5.143 


2.143 


p<.0001 


conflict 


1.049 


1.315 


1.236 


Self-esteem 


2.905 


5.333 


2.429 


p<.0001 




.944 


1.39 


1.076 


Setting goals 


2.857 


4.857 


2.0 


p<.0001 




.727 


1.315 


1.225 


Commitment 


3.143 


4.857 


1.714 


p<.0001 




1.276 


1.315 


1.007 


Responsibility 


3.19 


5.19 


2.0 


p<.0001 




1.169 


1.327 


1.095 


Initiative 


3.143 


4.714 


1.571 


p<.0001 




1.1-95 


1.347 


.811 


Paper work 


2.429 


4.714 


2.286 


p<.0001 




.87 


1.707 


1.309 


Machine setting 


3.238 


5.095 


1.857 


p<.0001 




1.179 


1.546 


1.276 



* In each pair of values, mean Is above and standard deviation is below. 
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